Pexip Infinity

Server Design Guide

Software Version 23.1

Document Version 23.a

March 2020
 Pexip Infinity Server Design Guide

Contents

Introduction 4

Summary of recommendations 5
Terminology 5
Management Node 5
Recommended host server specifications 5
Conferencing Node 6
Recommended host server specifications 6
Transcoding Conferencing Nodes versus Proxying Edge Nodes 6
General deployment recommendations for Transcoding Conferencing Nodes 7

Example Conferencing Node server configurations 9

Recommended server sizes 9
Other processor examples 9

Appendix 1: Detailed server hardware requirements 11

Host server hardware requirements 11
Capacity 12
Performance considerations 12
Intel AVX2 processor instruction set 12
AMD processors 12
Memory configuration 13
Example - dual socket, 4 channels 13
Example - dual socket, 6 channels 13

Appendix 2: Achieving high density deployments with NUMA 15

About NUMA 15
Conferencing Nodes and NUMA nodes 15
NUMA affinity and hyperthreading 16
Step-by-step guides 16
Summary of deployment recommendations 17

Appendix 3: VMware NUMA affinity and hyperthreading 18

Prerequisites 18
Overview of process 19
Setting NUMA affinity 19
Increasing vCPUs 21
Count logical processors 21
Assign vCPU and RAM 21

© 2020 Pexip AS Version 23.a March 2020 Page 2 of 29

 Pexip Infinity Server Design Guide

Reboot 22
Viewing updated capacity 22
Checking for warnings 23
BIOS settings 24
VMware and NUMA 24

Appendix 4: Hyper-V NUMA affinity and hyperthreading 25

Prerequisites 25
Example hardware 26
Disabling NUMA spanning on the server 27
Disable NUMA spanning on the VM 28
Starting the Virtual Machine 28
Viewing performance and checking for warnings 29
Moving VMs 29
BIOS settings 29

© 2020 Pexip AS Version 23.a March 2020 Page 3 of 29

 Pexip Infinity Server Design Guide Introduction

Introduction
This document describes the recommended specifications and deployment for servers hosting the Pexip Infinity platform, which
apply to both on-premises hardware and cloud deployments. It starts with a Summary of recommendations and some Example
Conferencing Node server configurations, which are supplemented by further details and explanations in the following
Appendices:

l Appendix 1: Detailed server hardware requirements provides a more detailed breakdown of the minimum and recommended
hardware requirements for servers hosting the Management Node and Conferencing Nodes respectively.
l Appendix 2: Achieving high density deployments with NUMA provides details on NUMA architecture and how this impacts
server architecture and overall performance of Conferencing Nodes.
l Appendix 3: VMware NUMA affinity and hyperthreading is for administrators with advanced VMware knowledge. It explains
how to experiment with VMware NUMA affinity and make use of hyperthreading for Pexip Infinity Conferencing Node VMs, in
order to achieve up to 50% additional capacity.
l Appendix 4: Hyper-V NUMA affinity and hyperthreading is for administrators with advanced Hyper-V knowledge. It explains
how to experiment with Hyper-V NUMA affinity and make use of hyperthreading for Pexip Infinity Conferencing Node VMs, in
order to achieve up to 50% additional capacity.

© 2020 Pexip AS Version 23.a March 2020 Page 4 of 29

 Summary of recommendations
Pexip Infinity Server Design Guide Terminology

Summary of recommendations
This section summarizes the terminology, recommended specifications and deployment guidelines for servers hosting the Pexip
Infinity platform. These apply to both on-premises hardware and cloud deployments.

Terminology
The table below provides descriptions for the terms used in this guide, in the context of a Pexip Infinity deployment.

Term Description

Processor The hardware within a computer that carries out the basic computing functions. It can consist of multiple cores.

Core One single physical processing unit. Intel Xeon E5 typically has 8 cores (10, 12 or more in newer versions)

Socket The socket on the motherboard where one processor is installed.

RAM Also referred to as "memory modules". The hardware that stores data which is accessed by the processor core while
executing programs.

Virtual CPU The VM's understanding of how many CPU cores it requires. Each vCPU appears as a single CPU core to the guest
(vCPU) operating system.

When configuring a Conferencing Node, you are asked to enter the number of virtual CPUs to assign to it. We
recommend no more than one virtual CPU per physical core, unless you are making use of CPUs that support
hyperthreading.

NUMA node The combination of a processor (consisting of one or more cores) and its associated memory.

Management Node

Recommended host server specifications

l 4 cores* (most modern processors will suffice)
l 4 GB RAM*
l 100 GB SSD storage
l The Pexip Infinity VMs are delivered as VM images (.ova etc.) to be run directly on the hypervisor. No OS should be installed.

* Sufficient for deployments of up to 30 Conferencing Nodes. For larger deployments, you will need to increase the amount of RAM
and number of cores. For guidance on Management Node sizing, consult your Pexip authorized support representative or your Pexip
Solution Architect.

© 2020 Pexip AS Version 23.a March 2020 Page 5 of 29

 Summary of recommendations
Pexip Infinity Server Design Guide Conferencing Node

Conferencing Node
Below are our general recommendations for Conferencing Node (Proxying Edge Nodes and Transcoding Conferencing Nodes) servers.
For some specific examples, see Example Conferencing Node server configurations.

Recommended host server specifications

l AVX or later instruction set required; we prefer AVX2 or AVX512. We recommend 2nd generation Intel Xeon Scalable
Processors (Cascade Lake) Gold 62xx or 52xx. We also support Intel Xeon Scalable Processors (Skylake) Gold 61xx generation or
E5-2600 v3/v4 Haswell/Broadwell architecture from 2014 or later. Also works with Xeon E5-2600 v1/v2 processors (Sandy
Bridge/Ivy Bridge from 2012 or later). AMD processors that support the AVX and AVX2 instruction set are also supported.
l 2.3 GHz (or faster) clock speed
l 10-20 physical cores per socket
l 1 GB RAM for each vCPU that is allocated to the Conferencing Node
l Populate all memory channels; there will be 4 on Xeon E5 processors (Sandy Bridge/Ivy Bridge/Haswell/Broadwell) and 6 on
Xeon Scalable processors (Skylake/Cascade Lake)
l Storage: 50 GB minimum per Conferencing Node; 500 GB total per server (to allow for snapshots etc.); SSD recommended
l RAID 1 mirrored storage
l Hypervisors: VMware ESXi 6.x; KVM; Xen; Microsoft Hyper-V 2012 or later
l The Pexip Infinity VMs are delivered as VM images (.ova etc.) to be run directly on the hypervisor. No OS should be installed.

Transcoding Conferencing Nodes versus Proxying Edge Nodes

The specifications and guidelines shown below apply to Transcoding Conferencing Nodes.
The servers hosting Proxying Edge Nodes do not require as high a specification as those servers hosting Transcoding Conferencing
Nodes. This is because proxying nodes are not as processor intensive as transcoding nodes. The minimum functional CPU instruction
set for a proxying node is AVX, which was first available in the Sandy Bridge generation. You still need multiple proxying nodes for
resilience and capacity.
We recommend allocating 4 vCPU and 4 GB RAM (which must both be dedicated resource) to each Proxying Edge Node, with a
maximum of 8 vCPU and 8 GB RAM for large or busy deployments.

© 2020 Pexip AS Version 23.a March 2020 Page 6 of 29

 Summary of recommendations
Pexip Infinity Server Design Guide Conferencing Node

General deployment recommendations for Transcoding Conferencing Nodes

Cores, CPU and RAM
l Prefer processors with high core count (10 cores or more per CPU).
l Prefer processors with a high clock speed (2.3 GHz and higher).
l Prefer a smaller number of large Conferencing Nodes (e.g. 4 x 10-core nodes), rather than large number of small Conferencing
Nodes (e.g. 10 x 4-core nodes).
l A single Conferencing Node must not be assigned more vCPU than the amount of physical cores on a single processor socket.
(An exception to this rule is when NUMA affinity is enabled.)
l For each physical CPU core (or logical thread, if employing NUMA affinity):
o configure 1 vCPU
o assign at least 1 GB RAM
For example, on an E5-2680v2 CPU with 10 physical cores (i.e. 20 logical threads) per CPU, either
o assign 10 vCPU (one per physical core) and 10 GB of RAM, or
o enable NUMA affinity, and assign 20 vCPU (one per logical thread) and 20 GB of RAM
l A Conferencing Node must have 4 vCPU and 4 GB RAM as an absolute minimum, and a maximum of 40 vCPU.
l Do not over-commit either RAM or CPU resources on hardware hosts. In other words, the Conferencing Node and Management
Node each must have dedicated access to their own RAM and CPU cores. Pexip Conferencing Nodes use real-time media, which
needs dedicated capacity.
l We recommend 8 memory modules for a dual E5-2600 configuration, as each CPU has 4 memory channels. 8 x 4 GB should be
sufficient for such deployments as we recommend 1 GB RAM per vCPU. Some vendors do not provide modules smaller than 8
GB, so in that case we suggest 8 x 8 GB. (This is more than required, but it could be useful if the server is repurposed in the
future.)
l Populate memory equally across all NUMA nodes/sockets on a single host server. All memory channels (typically 4 per CPU for
E5-2600; 6 per CPU for Gold 61xx) must be populated.
l For high performance clusters dedicated to Pexip Infinity, you can achieve 30-50% additional performance by using NUMA
affinity and taking advantage of hyperthreading (for CPUs supporting this). For more information, see NUMA affinity and
hyperthreading .

Hyperthreading
l Hyperthreading (also referred to as Hyper-Threading Technology), if supported, should always be left enabled by default.

BIOS performance settings

l Ensure all BIOS settings pertaining to power saving are set to maximize performance rather than preserve energy. (Setting these
to an energy-preserving or balanced mode may impact transcoding capacity, thus reducing the total number of HD calls that can
be provided.) The actual settings depend on the hardware vendor; some examples are given below:
Typical HP settings
o HP Power Profile: Maximum Performance
o Power Regulator modes: HP Static High Performance mode
o Energy/Performance Bias: Maximum Performance
o Memory Power Savings Mode: Maximum Performance
Typical Dell settings
o System Profile: Performance Optimized
Typical Cisco UCS B-Series settings
o System BIOS Profile (Processor Configuration) - CPU Performance: Enterprise
o System BIOS Profile (Processor Configuration) - Energy Performance: Performance
o VMware configuration: Active Policy: Balanced

© 2020 Pexip AS Version 23.a March 2020 Page 7 of 29

 Summary of recommendations
Pexip Infinity Server Design Guide Conferencing Node

Other BIOS settings (where available)

l Disable Sub-NUMA Clustering (SNC) (in this example via Chipset Configuration > North Bridge > UPI Configuration > Advanced
>SNC):

Network
l Although the Conferencing Node server will normally not use more than 1-2 Mbps per video call, we recommend 1 Gbps
network interface cards or switches to ensure free flow of traffic between Pexip Infinity nodes in the same datacenter. We do
not recommend 100 Mbps NIC.
l Redundancy: for hypervisors that support NIC Teaming (including VMware), you can configure two network interfaces for
redundancy, connected to redundant switches (if this is available in your datacenter).

Disk
l Although Pexip Infinity will work with SAS drives, we strongly recommend SSDs for both the Management Node and
Conferencing Nodes. General VM processes (such as snapshots and backups) and platform upgrades will be faster with SSDs.
l Management Node and Conferencing Node disks should be Thick Provisioned.
l Pexip Infinity can absorb and recover relatively gracefully from short bursts of I/O latency but sustained latency will create
problems.
l The Management Node requires a minimum of 800 IOPs (but we recommend providing more wherever possible).
l A Conferencing Node requires a minimum of 150 IOPs (but we recommend providing more wherever possible).
l Deployment on SAN/NAS storage should in most cases work well. Disk access is only required by the operating system and logs,
so a normal fair performance is expected.
l Redundancy: for RAID 1 mirroring for disk redundancy, remember to use a RAID controller supported by VMware or your
preferred hypervisor. The RAID controller must have an enabled cache. Most vendors can advise which of the RAID controllers
they provide are appropriate for your hypervisors.

Power
l Sufficient power to drive the CPUs. The server manufacturer will typically provide guidance on this.
l Redundancy: Dual PSUs.

© 2020 Pexip AS Version 23.a March 2020 Page 8 of 29

 Example Conferencing Node server configurations
Pexip Infinity Server Design Guide Recommended server sizes

Example Conferencing Node server configurations

This section provides some example server configurations for Transcoding Conferencing Nodes, with a capacity estimate of how
many HD and SD connections you can expect to achieve with each option. The lower figure in the range represents normal
configuration, and the upper figure represents what you can expect with NUMA affinity enabled. Note that the number of
connections are also sometimes referred to as ports.

Recommended server sizes

For the Pexip Infinity platform, the following server configurations provide maximum performance for cost:

Capacity (no. of connections)

110-140 HD / 220-280 SD 100-132 HD / 200-280 SD 55-75 HD / 100-150 SD

Cores / 2x20-core Cascade Lake 2x20-core Skylake 2x14-core Broadwell

Generation (2nd generation Xeon Scalable (Xeon Scalable Processors, launched (E5-2600 v4 generation, launched 2016)
Processors, launched 2019) 2017)

CPU 2 x Intel Xeon Gold 6230N 2 x Intel Xeon Gold 6148 2 x Intel E5-2680v4
l 20 core l 20 core l 14 core
l 2.3 GHz l 2.4 GHz l 2.4 GHz
l 27.5 MB cache l 27.5 MB cache l 35 MB cache

RAM 12 x 8 GB (6 RAM modules per CPU) 12 x 8 GB (6 RAM modules per CPU) 8 x 8 GB (minimum 4 RAM modules per
CPU for max memory bandwidth)

Network 1 Gbps NIC (we recommend dual NIC for redundancy)

Storage l 2 x 50 GB (50 GB minimum per Conferencing Node)

l 500 GB total per server (to allow for snapshots etc.)
l RAID 1 mirror for redundancy
l SSD recommended

Power We recommend redundant power

Example 1U l Cisco UCS C220 M5 l Cisco UCS C220 M5 l Cisco UCS C220 M4
rack l Dell R640 l Dell R640 l Dell R430/R630
servers
l HPE ProLiant DL360 Gen10 l HPE ProLiant DL360 Gen10 l HPE ProLiant DL360 Gen9
l Lenovo ThinkSystem SR650 l Lenovo ThinkSystem SR630 l IBM x3550 M5
l Supermicro 6019P l Supermicro 6019P l Supermicro 6018R-WTR

Other processor examples

With specifications similar to the above table, using other processors you can expect to achieve the following capacity:

HD connections Processor Cores Speed

50-88 2 x Intel E5-2695v4 18 2.1 GHz

45-70 2 x Intel E5-2690v3 12 2.6 GHz

50-60 2 x Intel E5-2680v3 12 2.5 GHz

© 2020 Pexip AS Version 23.a March 2020 Page 9 of 29

 Example Conferencing Node server configurations
Pexip Infinity Server Design Guide Other processor examples

HD connections Processor Cores Speed

40-50 2 x Intel E5-2680v2 10 2.8 GHz

36-55 2 x Intel E5-2660v3 10 2.6 GHz

36-52 2 x Intel E5-2630v4 10 2.2 GHz

32-50 2 x Intel E5-2650v3 10 2.3 GHz

© 2020 Pexip AS Version 23.a March 2020 Page 10 of 29

 Appendix 1: Detailed server hardware requirements
Pexip Infinity Server Design Guide Host server hardware requirements

Appendix 1: Detailed server hardware requirements

Host server hardware requirements

The following table lists the recommended hardware requirements for the Management Node and Conferencing Node (Proxying
Edge Nodes and Transcoding Conferencing Nodes) host servers.

Management Node Conferencing Node ††

Server manufacturer Any Any

Processor make Any We recommend 2nd generation Intel Xeon Scalable

Processors (Cascade Lake) Gold 62xx or 52xx. We also support
(see also Performance
Intel Xeon Scalable Processors (Skylake) Gold 61xx generation
considerations)
or E5-2600 v3/v4 Haswell/Broadwell architecture from 2014 or
later. Also works with Xeon E5-2600 v1/v2 processors (Sandy
Bridge/Ivy Bridge from 2012 or later). AMD processors that
support the AVX and AVX2 instruction set are also supported.

Processor instruction set Any AVX2 (AVX or later is supported) ‡

Processor architecture 64-bit 64-bit

Processor speed 2.0 GHz 2.3 GHz or faster

No. of physical cores * 4† 10-20 cores per socket

Processor cache no minimum 20 MB or greater

Total RAM * 4 GB† 1 GB RAM per vCPU, so either:

l 1 GB RAM per physical core (if deploying 1 vCPU per
core), or
l 2 GB RAM per physical core (if using hyperthreading and
NUMA affinity to deploy 2 vCPUs per core).

RAM makeup Any All channels must be populated with a DIMM, see Memory
configuration below. (For example, E5-24xx supports 3
DIMMs per socket, E5-26xx supports 4 DIMMs per socket.)

Hardware allocation The host server must not be over-committed in terms of either RAM or CPU. In other words, the
Management Node and Conferencing Nodes each must have dedicated access to their own RAM and
CPU cores.

Storage space required 100 GB SSD l 50 GB minimum per Conferencing Node

l 500 GB total per server (to allow for snapshots etc.)

Although Pexip Infinity will work with SAS drives, we strongly recommend SSDs for both the
Management Node and Conferencing Nodes. General VM processes (such as snapshots and backups)
and platform upgrades will be faster with SSDs.

GPU No specific hardware cards or GPUs are required.

Network Gigabit Ethernet connectivity from the host server.

Operating System The Pexip Infinity VMs are delivered as VM images (.ova etc.) to be run directly on the hypervisor. No OS
should be installed.

© 2020 Pexip AS Version 23.a March 2020 Page 11 of 29

 Appendix 1: Detailed server hardware requirements
Pexip Infinity Server Design Guide Capacity

Management Node Conferencing Node ††

Hypervisor l VMware ESXi 5.x or 6.x

(see also Performance l Microsoft Hyper-V 2012 or 2016

considerations) l Xen 4.2 or later
l KVM (Linux kernel 3.10.0 or later, and QEMU 1.5.0 or later)

* This does not include the processor and RAM requirements of the hypervisor.

† Sufficient for deployments of up to 30 Conferencing Nodes. For larger deployments, you will need to increase the amount of RAM and
number of cores. For guidance on Management Node sizing, consult your Pexip authorized support representative or your Pexip Solution
Architect.

‡ For VMware platforms, ESXi 6.x is required to make full use of the AVX2 instruction set. Note that AVX or later is required; older instruction
sets are not supported.

†† The servers hosting Proxying Edge Nodes do not require as high a specification as those servers hosting Transcoding Conferencing Nodes.
This is because proxying nodes are not as processor intensive as transcoding nodes. The minimum functional CPU instruction set for a
proxying node is AVX, which was first available in the Sandy Bridge generation. You still need multiple proxying nodes for resilience and
capacity. We recommend allocating 4 vCPU and 4 GB RAM (which must both be dedicated resource) to each Proxying Edge Node, with a
maximum of 8 vCPU and 8 GB RAM for large or busy deployments.

Capacity
The number of calls (or ports) that can be achieved per server in a Pexip Infinity deployment will depend on a number of things
including the specifications of the particular server and the bandwidth of each call.
As a general indication of capacity:

l When deployed on our recommended hardware (Intel Haswell, 10 cores, 2.3 GHz), Pexip Infinity can connect up to two High
Definition 720p30 calls per CPU core. This is based on 1.1 GHz per HD call plus 20% headroom. Capacity for higher speeds can
be linearly calculated based on these figures.
l The same recommended hardware can connect a higher number of lower-resolution calls per CPU core. For example, up to 20
audio-only AAC-LD calls at 64 kbps.
l Servers that are older, have slower processors, or have fewer CPUs, will have a lower overall capacity. Newer servers with faster
processors will have a greater capacity. Use of NUMA affinity and hyperthreading will also significantly increase capacity.

Performance considerations
The type of processors and Hypervisors used in your deployment will impact the levels of performance you can achieve. Some
known performance considerations are described below.

Intel AVX2 processor instruction set

As from software version 11, Pexip Infinity can make full use of the AVX2 instruction set provided by modern Intel processors. This
increases the performance of video encoding and decoding. For VMware platforms, ESXi 6.x is required to enable this optimization.
The VP9 codec is also available for connections to Conferencing Nodes running the AVX2 or later instruction set. VP9 uses around
one third less bandwidth for the same resolution when compared to VP8. Note however that VP9 calls consume around 1.5 times
the CPU resource (ports) on the Conferencing Node.

AMD processors
We have observed during internal testing that use of AMD processors results in a reduction of capacity (measured by ports per core)
of around 40% when compared to an identically configured Intel platform. This is because current AMD processors do not execute
advanced instruction sets at the same speed as Intel processors.
AMD processors older than 2012 may not perform sufficiently and are not recommended for use with the Pexip Infinity platform.

© 2020 Pexip AS Version 23.a March 2020 Page 12 of 29

 Appendix 1: Detailed server hardware requirements
Pexip Infinity Server Design Guide Memory configuration

Memory configuration
Memory must be distributed on the different memory channels (i.e. 4 channels per socket on the Xeon E5-2600; 6 channels per
socket on the Xeon Gold 61xx series).
There must be an equal amount of memory per socket, and all sockets must have all memory channels populated (you do not need
to populate all slots in a channel, one DIMM per channel is sufficient). Do not, for example, use two large DIMMs rather than four
lower-capacity DIMMs — using only two per socket will result in half the memory bandwidth, since the memory interface is
designed to read up from four DIMMs at the same time in parallel.

Example - dual socket, 4 channels

Xeon E5-2600 dual socket system:

l Each socket has 4 channels

l All 4 channels must be populated with a DIMM
l Both sockets must have the same configuration

Therefore for a dual socket E5-2600 you need 8 identical memory DIMMs.

Example - dual socket, 6 channels

Xeon Gold 61xx dual socket system:

l Each socket has 6 channels

l All 6 channels must be populated with a DIMM
l Both sockets must have the same configuration

Therefore for a dual socket Gold 61xx you need 12 or 24 identical memory DIMMs.

© 2020 Pexip AS Version 23.a March 2020 Page 13 of 29

 Appendix 1: Detailed server hardware requirements
Pexip Infinity Server Design Guide Memory configuration

© 2020 Pexip AS Version 23.a March 2020 Page 14 of 29

 Appendix 2: Achieving high density deployments with NUMA
Pexip Infinity Server Design Guide About NUMA

Appendix 2: Achieving high density deployments with NUMA

There are many factors that can affect the performance of Virtual Machines (VMs) running on host hardware. One of these is how
the VM interacts with NUMA (non-uniform memory access).
This section provides an overview of NUMA and how it applies to Pexip Infinity Conferencing Nodes. It summarizes our
recommendations and suggests best practices for maximizing performance.

About NUMA
NUMA is an architecture that divides the computer into a number of nodes, each containing one or more processor cores and
associated memory. A core can access its local memory faster than it can access the rest of the memory on that machine. In other
words, it can access memory allocated to its own NUMA node faster than it can access memory allocated to another NUMA node on
the same machine.
The diagram (right) outlines the physical
components of a host server and shows the
relationship to each NUMA node.

Conferencing Nodes and

NUMA nodes
We strongly recommend that a Pexip Infinity
Conferencing Node VM is deployed on a
single NUMA node to avoid the loss of
performance incurred when a core accesses
memory outside its own node.
In practice, with modern servers, each
socket represents a NUMA node. We
therefore recommend that:

l one Pexip Infinity Conferencing Node

VM is deployed per socket of the host
server
l the number of vCPUs that the Conferencing Node VM is configured to use is the same as or less than the number of physical
cores available in that socket (unless you are taking advantage of hyperthreading to deploy one vCPU per logical thread — in
which case see NUMA affinity and hyperthreading ).

© 2020 Pexip AS Version 23.a March 2020 Page 15 of 29

 Appendix 2: Achieving high density deployments with NUMA
Pexip Infinity Server Design Guide NUMA affinity and hyperthreading

This second diagram shows how the

components of a Conferencing Node virtual
machine relate to the server components
and NUMA nodes.
You can deploy smaller Conferencing Nodes
over fewer cores/threads than are available
in a single socket, but this will reduce
capacity.
Deploying a Conferencing Node over more
cores (or threads when pinned) than
provided by a single socket will cause loss of
performance, as and when remote memory
is accessed. This must be taken into account
when moving Conferencing Node VMs
between host servers with different
hardware configuration: if an existing VM is
moved to a socket that contains fewer
cores/threads than the VM is configured to
use, the VM will end up spanning two sockets and therefore NUMA nodes, thus impacting performance.
To prevent this occurring, ensure that either:

l you only deploy Conferencing Nodes on servers with a large number of cores per processor
l the number of vCPUs used by each Conferencing Node is the same as (or less than) the number of cores/threads available on
each NUMA node of even your smallest hosts.

NUMA affinity and hyperthreading

You can utilize the logical threads of a socket (hyperthreading) to deploy a Conferencing Node VM with two vCPUs per physical core
(i.e. one per logical thread) to achieve up to 50% additional capacity.
However, if you do this you must ensure that all Conferencing Node VMs are pinned to their respective sockets within the
hypervisor (also known as NUMA affinity). Otherwise, the Conferencing Node VMs will end up spanning multiple NUMA nodes,
resulting in a loss of performance.
Affinity does NOT guarantee or reserve resources, it simply forces a VM to use only the socket you define, so mixing Pexip
Conferencing Node VMs that are configured with NUMA affinity together with other VMs on the same server is not recommended.
NUMA affinity is not practical in all data center use cases, as it forces a given VM to run on a certain CPU socket (in this example),
but is very useful for high-density Pexip deployments with dedicated capacity.
NUMA affinity for Pexip Conferencing Node VMs should only be used if the following conditions apply:

l The server/blade is used for Pexip Conferencing Node VMs only, and the server will have only one Pexip Conferencing Node VM
per CPU socket (or two VMs per server in a dual socket CPU e.g. E5-2600 generation).
l vMotion (VMware) or Live Migration (Hyper-V) is NOT used. (Using these may result in having two nodes both locked to a single
socket, meaning both will be attempting to access the same processor, with neither using the other processor.)
l You fully understand what you are doing, and you are happy to revert back to the standard settings, if requested by Pexip
support, to investigate any potential issues that may result.

Step-by-step guides
For instructions on how to achieve NUMA pinning (also known as NUMA affinity) for your particular hypervisor, see:

l Appendix 3: VMware NUMA affinity and hyperthreading

l Appendix 4: Hyper-V NUMA affinity and hyperthreading

© 2020 Pexip AS Version 23.a March 2020 Page 16 of 29

 Appendix 2: Achieving high density deployments with NUMA
Pexip Infinity Server Design Guide Summary of deployment recommendations

Summary of deployment recommendations

We are constantly optimizing our use of the host hardware and expect that some of this advice may change in later releases of our
product. However our current recommendations are:

l Prefer processors with a high core count.

l Prefer a smaller number of large Conferencing Nodes rather than a larger number of smaller Conferencing Nodes.
l Deploy one Conferencing Node per NUMA node (i.e. per socket).
l Configure one vCPU per physical core on that NUMA node (without hyperthreading and NUMA pinning), or one vCPU per
logical thread (with hyperthreading and all VMs pinned to a socket in the hypervisor).
l Populate memory equally across all NUMA nodes on a single host server.
l Do not over-commit resources on hardware hosts.

© 2020 Pexip AS Version 23.a March 2020 Page 17 of 29

 Appendix 3: VMware NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Prerequisites

Appendix 3: VMware NUMA affinity and hyperthreading

This topic explains how to experiment with VMware NUMA affinity and Hyper-Threading Technology for Pexip Infinity Conferencing
Node VMs, in order to achieve up to 50% additional capacity.
If you are taking advantage of hyperthreading to deploy two vCPUs per physical core (i.e. one per logical thread), you must first
enable NUMA affinity; if you don't, the Conferencing Node VM will end up spanning multiple NUMA nodes, resulting in a loss of
performance.
Affinity does NOT guarantee or reserve resources, it simply forces a VM to use only the socket you define, so mixing Pexip
Conferencing Node VMs that are configured with NUMA affinity together with other VMs on the same server is not recommended.
NUMA affinity is not practical in all data center use cases, as it forces a given VM to run on a certain CPU socket (in this example),
but is very useful for high-density Pexip deployments with dedicated capacity.
This information is aimed at administrators with a strong understanding of VMware, who have very good control of their VM
environment, and who understand the consequences of conducting these changes.
Please ensure you have read and implemented our recommendations in Appendix 2: Achieving high density deployments with
NUMA before you continue.

Prerequisites
You must be using the VMware vSphere Flash-based web client in order to perform this configuration.
VMware NUMA affinity for Pexip Conferencing Node VMs should only be used if the following conditions apply:

l The server/blade is used for Pexip Conferencing Node VMs only, and the server will have only one Pexip Conferencing Node VM
per CPU socket (or two VMs per server in a dual socket CPU e.g. E5-2600 generation).
l vMotion (VMware) or Live Migration (Hyper-V) is NOT used. (Using these may result in having two nodes both locked to a single
socket, meaning both will be attempting to access the same processor, with neither using the other processor.)
l You fully understand what you are doing, and you are happy to revert back to the standard settings, if requested by Pexip
support, to investigate any potential issues that may result.

Example server without NUMA affinity - allows for more mobility of VMs

© 2020 Pexip AS Version 23.a March 2020 Page 18 of 29

 Appendix 3: VMware NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Overview of process

Example server with NUMA affinity - taking advantage of hyperthreading to gain 30-50% more capacity per server

Overview of process
We will configure the two Conferencing Node VMs (in this example, an E5-2600 CPU with two sockets per server) with the following
advanced VMware parameters:

Conferencing Node A locked to Socket 0

l cpuid.coresPerSocket = 1
l numa.vcpu.preferHT = TRUE
l numa.nodeAffinity = 0

Conferencing Node B locked to Socket 1

l cpuid.coresPerSocket = 1
l numa.vcpu.preferHT = TRUE
l numa.nodeAffinity = 1

You must also double-check the flag below to ensure it matches the number of vCPUs in the Conferencing Node:

l numa.autosize.vcpu.maxPerVirtualNode

For example, it should be set to 24 if that was the number of vCPUs you assigned.

Setting NUMA affinity

Before you start, please consult your local VMware administrator to understand whether this is appropriate in your
environment.
1. Shut down the Conferencing Node VMs, to allow you to edit their settings.
2. Give the Conferencing Node VMs names that indicate that they are locked to a given socket (NUMA node). In the example
below the VM names are suffixed by numa0 and numa1:

3. Right-click the first Conferencing Node VM in the inventory and select Edit Settings.
4. From the VM Options tab, expand the Advanced section and select Edit Configuration:

© 2020 Pexip AS Version 23.a March 2020 Page 19 of 29

 Appendix 3: VMware NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Setting NUMA affinity

5. At the bottom of the window that appears, enter the following Names and corresponding Values for the first VM, which should
be locked to the first socket (numa0):
o cpuid.coresPerSocket = 1
o numa.vcpu.preferHT = TRUE
o numa.nodeAffinity = 0
It should now look like this in the bottom of the parameters list:

6. Select OK and OK again.

Now our conf-node_numa0 Virtual Machine is locked to numa0 (the first socket).
7. Repeat the above steps for the second node, entering the following data for the second VM, which should be locked to the
second socket (numa1):
o cpuid.coresPerSocket = 1
o numa.vcpu.preferHT = TRUE
o numa.nodeAffinity = 1
It should now look like this in the bottom of the parameters list:

8. Select OK and OK again.

© 2020 Pexip AS Version 23.a March 2020 Page 20 of 29

 Appendix 3: VMware NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Increasing vCPUs

Now our conf-node_numa1 Virtual Machine is locked to numa1 (the second socket).

It is very important that you actually set numa.nodeAffinity to 1 and not 0 for the second node. If both are set to 0, you will
effectively only use numa node 0, and they will fight for these resources while leaving numa node 1 unused.

Increasing vCPUs
You must now increase the number of vCPUs assigned to your Conferencing Nodes, to make use of the hyperthreaded cores.
(Hyperthreading must always be enabled, and is generally enabled by default.)

Count logical processors

First you must check how many logical processors each CPU has.
In the example screenshot below, the E5-2680 v3 CPU has 12 physical cores per CPU socket, and there are two CPUs on the server.

With hyperthreading, each physical core has 2 logical processors , so the CPU has 24 logical processors (giving us a total of 48 with
both CPUs).
In this case 2 x 12 = 24 is the "magic number" we are looking for with our Conferencing Nodes - which is double the amount of
Cores per Socket.

Assign vCPU and RAM

Next, you must edit the settings on the Virtual Machines to assign 24 vCPU and 24 GB RAM to each of the two Conferencing Nodes.
Ensure that the server actually has 24 GB of RAM connected to each CPU socket. Since all four memory channels should be
populated with one RAM module each, you will normally require 4 x 8 GB per CPU socket.

© 2020 Pexip AS Version 23.a March 2020 Page 21 of 29

 Appendix 3: VMware NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Viewing updated capacity

Reboot
Finally, save and boot up your virtual machines. After about 5 minutes they should be booted, have performed their performance
sampling, and be available for calls.

Viewing updated capacity

To view the updated capacity of the Conferencing Nodes, log in to the Pexip Management Node, select Status > Conferencing
Nodes and then select one of the nodes you have just updated. The Maximum capacity - HD connections field should now show
slightly less than one HD call per GHz (compared to the previous one HD call per 1.41 GHz).
In our example, 12 physical cores x 2.6 GHz = 31.2 GHz, so the Conferencing Node should show around 30 or 31 HD calls, assuming a
balanced BIOS power profile. With maximum performance BIOS power profiles, the results could be up to 33-34 HD calls per
Conferencing Node VM.
Our first VM:

© 2020 Pexip AS Version 23.a March 2020 Page 22 of 29

 Appendix 3: VMware NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Checking for warnings

Our second VM:

Checking for warnings

You should check for warnings by searching the administrator log (History & Logs > Administrator Log) for " sampling".
A successful run of the above example should return something like:
2015-04-05T18:25:40.390+00:00 softlayer-lon02-cnf02 2015-04-05 18:25:40,389 Level="INFO" Name="administrator.system"
Message="Performance sampling finished" Detail="HD=31 SD=60 Audio=240"

An unsuccessful run, where VMware has split the Conferencing Node over multiple NUMA nodes, would return the following
warning in addition to the result of the performance sampling:
2015-04-06T17:42:17.084+00:00 softlayer-lon02-cnf02 2015-04-06 17:42:17,083 Level="WARNING" Name="administrator.system"
Message="Multiple numa nodes detected during sampling " Detail="We strongly recommend that a Pexip Infinity
Conferencing Node is deployed on a single NUMA node"
2015-04-06T17:42:17.087+00:00 softlayer-lon02-cnf02 2015-04-06 17:42:17,086 Level="INFO" Name="administrator.system"
Message="Performance sampling finished" Detail="HD=21 SD=42 Audio=168"

If you have followed the steps in this guide to set NUMA affinity correctly and you are getting the warning above, this could be due
to another VMware setting. From VMware, select the Conferencing Node and then select Edit Settings > Options > General >
Configuration Parameters...). The numa.autosize.vcpu.maxPerVirtualNode option should be set to your "magic number". For
example, 24 is our "magic number" - the number of logical processors, or vCPUs, assigned in our example.
If this option is set to anything lower, e.g. 8, 10 or 12, VMware will create two virtual NUMA nodes, even if locked on one socket.

© 2020 Pexip AS Version 23.a March 2020 Page 23 of 29

 Appendix 3: VMware NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide BIOS settings

BIOS settings
Ensure all BIOS settings pertaining to power saving are set to maximize performance rather than preserve energy. (Setting these to
an energy-preserving or balanced mode may impact transcoding capacity, thus reducing the total number of HD calls that can be
provided.) While this setting will use slightly more power, the alternative is to add another server in order to achieve the increase
in capacity, and that would in total consume more power than one server running in high performance mode.
The actual settings will depend on the hardware vendor; see BIOS performance settings for some examples.

A quick way to verify that BIOS has been set appropriately is to check the hardware's Power Management settings in VMware
(select the host then select Configure > Hardware > Power Management). In most cases, the ACPI C-states should not be exposed
to VMware when BIOS is correctly set to maximize performance.
If the ACPI C-states are showing in VMware (as shown below), the BIOS has most likely not been set to maximize performance :

When BIOS has been correctly set to maximize performance, it should in most cases look like this:

If your server is set to maximize performance, but VMware still shows ACPI C-states, change it to balanced (or similar), and then
change back to maximize performance. This issue has been observed with some Dell servers that were preconfigured with
maximize performance, but the setting did not take effect initially.

VMware and NUMA

As well as the physical restrictions discussed above, the hypervisor can also impose restrictions. VMware provides virtual NUMA
nodes on VMs that are configured with more than 8 CPUs. If you have fewer than 8 CPUs, you should change this default by setting
numa.vcpu.min in the VM's configuration file to the number of vCPUs you wish to configure (which will be double the number of
CPUs you have available).
For more information, see https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.resmgmt.doc/GUID-
3E956FB5-8ACB-42C3-B068-664989C3FF44.html.

© 2020 Pexip AS Version 23.a March 2020 Page 24 of 29

 Appendix 4: Hyper-V NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Prerequisites

Appendix 4: Hyper-V NUMA affinity and hyperthreading

This topic explains how to experiment with NUMA pinning and Hyper-Threading Technology for Pexip Infinity Conferencing Node
VMs, in order to achieve up to 50% additional capacity.
If you are taking advantage of hyperthreading to deploy two vCPUs per physical core (i.e. one per logical thread), you must first
enable NUMA affinity; if you don't, the Conferencing Node VM will end up spanning multiple NUMA nodes, resulting in a loss of
performance.
Affinity does NOT guarantee or reserve resources, it simply forces a VM to use only the socket you define, so mixing Pexip
Conferencing Node VMs that are configured with NUMA affinity together with other VMs on the same server is not recommended.
NUMA affinity is not practical in all data center use cases, as it forces a given VM to run on a certain CPU socket (in this example),
but is very useful for high-density Pexip deployments with dedicated capacity.
This information is aimed at administrators with a strong understanding of Hyper-V, who have very good control of their VM
environment, and who understand the consequences of conducting these changes.
Please ensure you have read and implemented our recommendations in Appendix 2: Achieving high density deployments with
NUMA before you continue.

Prerequisites
NUMA affinity for Pexip Conferencing Node VMs should only be used if the following conditions apply:

l The server/blade is used for Pexip Conferencing Node VMs only, and the server will have only one Pexip Conferencing Node VM
per CPU socket (or two VMs per server in a dual socket CPU e.g. E5-2600 generation).
l vMotion (VMware) or Live Migration (Hyper-V) is NOT used. (Using these may result in having two nodes both locked to a single
socket, meaning both will be attempting to access the same processor, with neither using the other processor.)
l You fully understand what you are doing, and you are happy to revert back to the standard settings, if requested by Pexip
support, to investigate any potential issues that may result.

Example server without NUMA affinity - allows for more mobility of VMs

© 2020 Pexip AS Version 23.a March 2020 Page 25 of 29

 Appendix 4: Hyper-V NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Example hardware

Example server with NUMA affinity - taking advantage of hyperthreading to gain 30-50% more capacity per server

Example hardware
In the example given below, we are using a SuperMicro SuperServer with dual Intel Xeon E5-2680-v3 processors, 64GB RAM, and 2
x 1TB hard drives.
On this server:

l we deploy one Conferencing Node VM per processor/socket, so two Conferencing Nodes in total
l we disable NUMA spanning, so each Conferencing Node VM runs on a single NUMA node/processor/socket
l each processor has 12 physical cores
l we use hyperthreading to deploy 2 vCPUs per physical core
l this gives us 24 vCPUs / 24 threads per Conferencing Node
l therefore we get 48vCPUs / 24 threads in total on the server.

© 2020 Pexip AS Version 23.a March 2020 Page 26 of 29

 Appendix 4: Hyper-V NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Disabling NUMA spanning on the server

Disabling NUMA spanning on the server

Firstly, we must disable NUMA spanning on the server. To do this:

1. From within Hyper-V Manager, right-click on the server and select Hyper-V Settings...:

2. From the Server section, select NUMA Spanning and disable Allow virtual machines to span physical NUMA nodes. This
ensures that all processing will remain on a single processor within the server:

© 2020 Pexip AS Version 23.a March 2020 Page 27 of 29

 Appendix 4: Hyper-V NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Disable NUMA spanning on the VM

Disable NUMA spanning on the VM

Next we need to ensure the Conferencing Node VMs have the correct settings too, and do not span multiple processors.
To do this:

1. From within Hyper-V, select the Conferencing Node VM, and then select Settings > Hardware > Processor > NUMA.
2. Confirm that only 1 NUMA node and 1 socket are in use by each Conferencing Node VM:

Starting the Virtual Machine

After the NUMA settings have been changed, you can start up each of the Conferencing Node VMs:

© 2020 Pexip AS Version 23.a March 2020 Page 28 of 29

 Appendix 4: Hyper-V NUMA affinity and hyperthreading
Pexip Infinity Server Design Guide Viewing performance and checking for warnings

Viewing performance and checking for warnings

Every time a Conferencing Node is started up or rebooted, the Pexip Infinity Management Node will perform a sampling of the
system to understand what capabilities it has. To view this information, go to the administrator log (History & Logs > Administrator
Log) and search for " sampling".
A successful run of the above example should return something like:
2015-04-05T18:25:40.390+00:00 softlayer-lon02-cnf02 2015-04-05 18:25:40,389 Level="INFO" Name="administrator.system"
Message="Performance sampling finished" Detail="FULLHD=17 HD=33 SD=74 Audio=296"

An unsuccessful run, where Hyper-V has split the Conferencing Node over multiple NUMA nodes, would return the following
warning in addition to the result of the performance sampling:
2015-04-06T17:42:17.084+00:00 softlayer-lon02-cnf02 2015-04-06 17:42:17,083 Level="WARNING" Name="administrator.system"
Message="Multiple numa nodes detected during sampling " Detail="We strongly recommend that a Pexip Infinity
Conferencing Node is deployed on a single NUMA node"
2015-04-06T17:42:17.087+00:00 softlayer-lon02-cnf02 2015-04-06 17:42:17,086 Level="INFO" Name="administrator.system"
Message="Performance sampling finished" Detail="HD=21 SD=42 Audio=168"

Moving VMs
When moving Conferencing Node VMs between hosts, you must ensure that the new host has at least the same number of cores.
You must also remember to disable NUMA spanning on the new host.

BIOS settings
Ensure all BIOS settings pertaining to power saving are set to maximize performance rather than preserve energy. (Setting these to
an energy-preserving or balanced mode may impact transcoding capacity, thus reducing the total number of HD calls that can be
provided.) While this setting will use slightly more power, the alternative is to add another server in order to achieve the increase
in capacity, and that would in total consume more than one server running in high performance mode.
The actual settings will depend on the hardware vendor; see BIOS performance settings for some examples.

© 2020 Pexip AS Version 23.a March 2020 Page 29 of 29