Metamux-3 6 0-Userguide
Metamux-3 6 0-Userguide
Release 3.6.0
www.arista.com
1 Introduction 1
1.1 Licenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Servicing & Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Specification 3
2.1 Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4 Overview 18
4.1 About MetaMux App . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2 Mux Modes naming convention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.3 MetaMux on Multi-FPGA platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.4 Supported Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.5 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.6 Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5 Configuration 24
5.1 Application interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.2 Enabling applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.3 MetaMux configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.4 Configuration examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.5 MetaMux operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
i
7 Telemetry 35
7.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7.2 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
12 Command Reference 47
12.1 Privileged mode commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
12.2 MetaMux Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
13 Appendix 58
13.1 Latency measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Index 113
ii
CHAPTER 1
Introduction
1.1 Licenses
A license to use MOS and other Arista software is described in the “Terms and Conditions for Sale of Products”
agreement.
Arista software also contains open-source that is covered by the GNU General Public License (GPL) and other
licences.
Copies of the licenses can be found in the /usr/share/common-licenses directory on your Arista 7130 device.
You may obtain the MOS source code covered under the GPL licences from Arista by sending a request to sup-
[email protected].
1
MetaMux App User Guide, Release 3.6.0
1.2 Scope
This manual describes the MetaMux application for the Arista 7130 devices.
1.3 Conventions
This manual covers many areas of operation of this product, however, should you have any questions, problems
or comments then please contact our support services:
www.arista.com
Email: [email protected]
For support requirements:
• Australia: Phone: +61-2-8310-4381
• North America: Phone: +1-408-547-5502, Toll Free: +1-866-476-0000
• United Kingdom: Phone: +44-207-023-9352 Toll Free: +44-808-234-0722
• Further numbers are listed on our Customer Support page
Important: For support problems, please make the following information available:
• the device serial number;
• a .zip file with all relevant device info (refer to the show tech-support command).
Arista hardware should not be serviced by the end-user. Please contact Arista support for assistance.
1.2. Scope 2
CHAPTER 2
Specification
2.1 Specification
• MetaMux 3.x application runs on the Arista 7130E and L-Series Platforms.
• Supports multiple configurations of ingress to egress multiplexing, which are dependent on platform and
FPGA capabilities.
• Packet scheduling can be configured via CLI to
– Order of Arrival, followed by lowest to highest ap port (per multiplexer) when coincident. (Default)
– Round Robin, lowest to highest ap port per multiplexer.
• Packet buffering can be disabled per multiplexer. When disabled, all packet collisions will result in dropped
packets (counted per ingress port).
• Approximately 16kBytes of packet buffering per ingress port.
• Throughput is greater than 94.011% based on Spirent Benchmark test result.
• Throughput of 100% is possible if adherence to the IEEE 802.3 10GE standard minimum inter-frame gap
(IFG) is disabled (can be disabled per mux). When disabled, all queued packets are transmitted back to back
with no ethernet idle block between packets, this can result in an IFG as small as zero bytes. This violates
the IEEE 802.3 10GE standard minimum inter-frame gap and some equipment will not ingest this traffic
without errors.
• Does not support jumbo frames. Frames over ~1600Bytes are truncated.
• Does not support packets less than 64Bytes (runt packets).
• Packet counters that include:
– Total received packets per ingress port
– Total packets transmitted per egress port
– Total dropped packets per ingress port (buffer overflow)
– Total packets detected with errors.
• Packet queuing statistics (Qstats) that include:
– Average Queuing Time
– Number of queued packets
– Largest Queuing Time
• All counters and Qstats are available through influx telemetry database. Refer to the MOS manual for
details.
3
MetaMux App User Guide, Release 3.6.0
2.2 Limitations
• Qstats is only availble in the main/central FPGA at this time (no leaf support).
• RCVT is only availble in the main/central FPGA at this time (no leaf support).
Note: Please refer to the release notes for current known issues
2.3 Roadmap
2.2. Limitations 4
CHAPTER 3
This release supports the e_central, e_leaf, eh_central, eh_leaf, l and lb2 board standards, each implemented by
several device SKUs.
Modes e_central e_leaf eh_central eh_leaf l lb2
4x12_pf Supported Unsupported Unsupported Unsupported Unsupported Unsupported
4x12 Supported Unsupported Supported Unsupported Supported Supported
2x8+2x16 Supported Unsupported Supported Unsupported Supported Supported
1x48 Supported Unsupported Supported Unsupported Supported Supported
1x12 Unsupported Supported Unsupported Supported Unsupported Unsupported
12x4 Supported Unsupported Supported Unsupported Supported Supported
1x16+1x32 Supported Unsupported Supported Unsupported Supported Supported
24x2 Supported Unsupported Supported Unsupported Supported Supported
6x8_eb Supported Unsupported Supported Unsupported Supported Supported
3x4 Unsupported Supported Unsupported Supported Unsupported Unsupported
6x8 Supported Unsupported Supported Unsupported Supported Supported
2x24 Supported Unsupported Supported Unsupported Supported Supported
2x8+2x16_eb Supported Unsupported Supported Unsupported Supported Supported
3.1 E_central
3.1.1 2x24
3.1.2 6x8_eb
5
MetaMux App User Guide, Release 3.6.0
3.1.3 12x4
3.1.4 1x48
3.1.5 4x12
3.1.6 1x16+1x32
3.1.7 4x12_pf
3.1. E_central 6
MetaMux App User Guide, Release 3.6.0
3.1.8 24x2
3.1.9 2x8+2x16
3.1.10 2x8+2x16_eb
3.1. E_central 7
MetaMux App User Guide, Release 3.6.0
3.1.11 6x8
3.2 E_leaf
3.2.1 3x4
Leaf_b
Leaf_a
3.2.2 1x12
Leaf_b
3.2. E_leaf 8
MetaMux App User Guide, Release 3.6.0
Leaf_a
3.3 Eh_central
3.3.1 6x8
3.3.2 1x48
3.3.3 1x16+1x32
3.3.4 4x12
3.3.5 12x4
3.3. Eh_central 9
MetaMux App User Guide, Release 3.6.0
3.3.6 2x24
3.3.7 6x8_eb
3.3.8 2x8+2x16_eb
3.3.9 2x8+2x16
3.3. Eh_central 10
MetaMux App User Guide, Release 3.6.0
3.3.10 24x2
3.4 Eh_leaf
3.4.1 3x4
Leaf_b
Leaf_a
3.4. Eh_leaf 11
MetaMux App User Guide, Release 3.6.0
3.4.2 1x12
Leaf_b
Leaf_a
3.5 L
3.5.1 6x8_eb
3.5.2 1x48
3.5.3 1x16+1x32
3.5. L 12
MetaMux App User Guide, Release 3.6.0
3.5.4 12x4
3.5.5 2x8+2x16_eb
3.5.6 24x2
3.5. L 13
MetaMux App User Guide, Release 3.6.0
3.5.7 4x12
3.5.8 2x24
3.5.9 6x8
3.5.10 2x8+2x16
3.6 Lb2
3.6.1 6x8_eb
3.6. Lb2 14
MetaMux App User Guide, Release 3.6.0
3.6.2 2x8+2x16_eb
3.6.3 24x2
3.6.4 2x8+2x16
3.6.5 2x24
3.6. Lb2 15
MetaMux App User Guide, Release 3.6.0
3.6.6 12x4
3.6.7 4x12
3.6.8 1x48
3.6.9 1x16+1x32
3.6.10 6x8
3.6. Lb2 16
MetaMux App User Guide, Release 3.6.0
3.6. Lb2 17
CHAPTER 4
Overview
MetaMux is an application (app) that runs on Arista’s 7130E and 7130L-Series platforms. It performs ultra-low-
latency multiplexing with and without packet contention queuing. The port to port latency is a function of the
particular mode selected, front panel ingress port, front panel egress port, AP ingress port and platform being
used. Detailed latency numbers along with a summary are provided in later sections.
Note: As of MetaMux 3.0, support for 7130C and 7130K-Series platforms has been removed. For these plat-
forms, please revert to MetaMux Version 2.
Multiplexing (also known as packet aggregation or muxing) involves the aggregation of multiple ingress con-
nections into one egress stream. For instance, multiple trading servers often need to communicate with a single
exchange at the same time, or the output of multiple taps might be aggregated into a single logging feed.
As packet aggregation must schedule packets arriving on multiple ingress ports to a single egress port, there is the
possibility of a collision occurring. In the case where there is a collision, packets can be selectively queued, or
dropped to force a minimum packet latency policy. The default transmission of packets strictly follows an order
of arrival policy, however this can be optionally changed to a round-robin via the CLI.
18
MetaMux App User Guide, Release 3.6.0
Currently the modes in MetaMux app are compliant with 10G Ethernet only. The MetaMux app supports a number
of configurations with different sized multiplexers and queueing options.
MetaMux can be configured to use any FPGA on supported the multi-FPGA 7130EP and 7130EH platforms.
Each FPGA in the system will support a given set of modes that are shown with the list fpga modes com-
mand. A mode is set to a given FPGA with the fpga FPGA_NAME mode MODE_NAME command, where the
FPGA_NAME and MODE_NAME come from from the list fpga modes command.
Once a mode is set for each FPGA, the number of muxes that will be made available are the concatenation of the
modes of each FPGA. For example, if the 6x8 mode is set in the central/default FPGA and a 3x4 mode is set in
the leaf-a FPGA then the global resulting mode will be a 6x8+3x4, i.e., six 8-to-1 multiplexers instantiated in
the central/default FPGA and three 4-to-1 multiplexers in the leaf-a/secondary FPGA.
Note that care should be taken to not set a mode on an FPGA that is being used by another application. MetaMux
will program the FPGAs that have a mode regardless of other applications currently running.
MetaMux 3.x no longer supports all modes from previous Metamux revisions (legacy).
MetaMux 3.x adds a new ultra-low-latency architecture for the 7130E and 7130L-Series platforms.
2x8+2x16_eb Two 8-to-1 and two 16-to-1 multiplexers with elastic buffer
1x16+1x32 One 16-to-1 and one 32-to-1 multiplexers
2x24 Two 24-to-1 multiplexers
24x2 Twenty Four 2-to-1 multiplexers
1x48 One 48-to-1 multiplexer
Additionally, the 7130E & EP platforms support the following custom mode:
Mode Description
------------ -------------------------------------------------------
4x12_pf Four 12-to-1 multiplexers (+ingress flow control)
The 7130EP & EH multi-FPGA platforms support the following modes on the “leaf” FPGAs:
Mode Description FPGA
------------ ---------------------------------- --------------------
1x12 One 12-to-1 multiplexer mezzanine.leaf_a
3x4 Three 4-to-1 multiplexers mezzanine.leaf_a
4.5 Latency
The measured latencies described here are a function of the mode, mezzanine, platform and configured port
interconnect through the cross-point matrix. This means that the latency of different combinations of FPGA ports
and external ports will result in slightly different results. Generally speaking, the external ports near the “center”
of the device have the lowest latency.
The latency measurements were collected using a 7130-48L device running MetaWatch directly connected to the
MetaMux device under test. The MetaWatch device then generates random packets that are sent to both a local
MetaWatch core input and the MetaMux device under test. The return path from the MetaMux device is then fed
to a second MetaWatch core input. The time difference between the two captured stream of packets (minus the
cable delay) is the measured latency through a MetaMux port, independent of other ports. The process is repeated
over all ports and modes of MetaMux independently of each other.
The external port used to send and receive packets to and from the MetaMux device is et1.
Cable delay is measured at the beginning of the process by connecting both ends of the cables in a loopback
fashion. This means that latencies shown here include the SFPs in the MetaMux device.
4.5. Latency 20
MetaMux App User Guide, Release 3.6.0
The following tables summarise the latencies through MetaMux measured in Arista’s laboratory as described
before. For each mux type, the average and minimum measured latency in nanoseconds is shown as well as the
device type and mode from where the numbers were gathered.
All modes and FPGAs have approximately 16kByte buffers on ingress.
The following tables summarise latency measurements for MetaMux 3.x on the 7130E, EP, EB & EH platforms
with a selection of front panel port counts. The tables below give the lowest recorded Min, the Avg of the port with
the lowest recorded Min and highest recorded Max latencies. The Mode column reflects the particular MetaMux
mode used to obtain the measurement.
4.5. Latency 21
MetaMux App User Guide, Release 3.6.0
The following tables summarise latency measurements for MetaMux 3.x on the 7130L & LB platforms with the
48 front panel port count. The tables below give the lowest recorded Min, the Avg of the port with the lowest
recorded Min and highest recorded Max latencies. The Mode column reflects the particular Mux mode used to
obtain the measurement.
4.6 Throughput
Based on the Spirent benchmark test RFC2544, the supported throughput of a single port is greater than 94.011%.
This is the same for all mode and platform configurations.
4.6. Throughput 22
MetaMux App User Guide, Release 3.6.0
4.6. Throughput 23
CHAPTER 5
Configuration
The number of externally facing ports is determined by the hosting platform. These ports appear in the user
interface list as et1 through etX, where X is 32, 48 or 96.
The number of internal application (FPGA) ports is determined by the mezzanine and the host platform. These
ports appear in the user interface list as ap1 through apX. The following table provides details of the various
combinations.
32 48 96
7130E-Series ap1-56 ap1-56 ap1-56
7130L-Series ap1-60 ap1-60 ap1-28,31-60
All of these ports can be configured using the Layer 1 (matrix) switch built into the host platform. For example,
to pass data between front panel ports et1 and et2 with the minimum latency (4 ns), the following MOS CLI
commands are used.
hostname> en
hostname> conf
hostname(config)> int et1
hostname(config-if-et1)> source et2
hostname(config-if-et1)> int et2
hostname(config-if-et2)> source et1
Application interfaces can be treated in the same way. To direct the output from the ap1 interface to front panel
port et1, the following commands are used:
hostname> en
hostname> conf
hostname(config)> int et1
hostname(config-if-et1)> source ap1
Similarly, to pass the data from front panel port et1 to the ap1 interface, the following commands are used:
hostname> en
hostname> conf
hostname(config)> int ap1
hostname(config-if-ap1)> source et1
Depending on the function of those application interfaces it is possible to create intricate routing networks using
the building blocks provided.
24
MetaMux App User Guide, Release 3.6.0
In order to start the MetaMux app it must be installed. Applications are installed in /opt/apps. Presently,
MetaMux 2.0 ships with the default MOS distribution. You will be required to upgrade MetaMux app independent
of MOS, using the following commands:
scp metamux-XXX.rpm admin@hostname:
hostname> en
hostname> conf
hostname(config)> install app file:metamux-XXX.rpm
hostname(config)> exit
hostname> exit
You can see the apps installed on your device using the “show version” command.
Enabling the MetaMux app is done by entering the metamux app mode, selecting a given FPGA mode and exe-
cuting “no shutdown”. For example:
hostname> en
hostname> conf
hostname(config)> app metamux
hostname(config-app-metamux)> fpga mezzanine.central mode 6x8
hostname(config-app-metamux)> no shutdown
Programming mezzanine.central FPGA...
hostname(config-app-metamux)>
The MetaMux app configures the device to behave as a multiplexer, but does not configure the internal matrix
switch. This must be done via the source, connect and dest commands for each interface. Descriptions are
added to each interface to make it clear what the function of each port is. Following the “no shutdown” operation
above, the MetaMux app shows the following.
hostname(config-if-ap1)> show int ap* desc
Port Description
---- -----------
ap1 Mux 0 In 0
ap2 Mux 0 In 1
ap3 Mux 0 In 2
ap4 Mux 0 In 3, Out
ap5 Mux 0 In 4
ap6 Mux 0 In 5
ap7 Mux 0 In 6
ap8 Mux 0 In 7
ap9 Mux 1 In 0
ap10 Mux 1 In 1
ap11 Mux 1 In 2
ap12 Mux 1 In 3, Out
ap13 Mux 1 In 4
ap14 Mux 1 In 5
ap15 Mux 1 In 6
ap16 Mux 1 In 7
ap17 Mux 2 In 0
ap18 Mux 2 In 1
ap19 Mux 2 In 2
ap20 Mux 2 In 3, Out
ap21 Mux 2 In 4
ap22 Mux 2 In 5
ap23 Mux 2 In 6
ap24 Mux 2 In 7
...
ap33 Mux 3 In 0
ap34 Mux 3 In 1
ap35 Mux 3 In 2
There are a number of different modes of operation for each FPGA available in the device. The FPGA mode is
set with the fpga NAME mode MODE command. The fpga NAME and corresponding mode are listed with the
list fpga modes command.
For backwards compatibility, a single “mode” command is available that is aliased to the main/central FPGA. On
devices with only one FPGA using either the fpga NAME mode or the single mode command is equivalent,
although it is recommended to use the former.
Note: It is the users responsibility to make sure that a mode is not being set for an FPGA that is in use by another
application as MetaMux will overwrite the configuration of all FPGAs that have a mode configured.
Using the mode command updates the behaviour of the application interfaces as well as their descriptions.
For example, setting the application to a 48-to-1 mux on the central FPGA and a 12-to-1 on the leaf_a mode can
be done as follows (leaf_a FPGA only available on certain platforms):
hostname> en
hostname> conf
hostname(config)> app metamux
hostname(config-app-metamux)> fpga mezzanine.central mode 1x48
hostname(config-app-metamux)> fpga mezzanine.leaf_a mode 1x12
hostname(config-app-metamux)> no shutdown
Programming FPGA mezzanine.central...
Programming FPGA mezzanine.leaf_a...
Configuring FPGAs...
Starting Qstats...
Starting Linkmon...
hostname(config-app-metamux)> show int ap* desc
Port Description
---- -----------
ap1 Mux 0 In 0
ap2 Mux 0 In 1
ap3 Mux 0 In 2
ap4 Mux 0 In 3
ap5 Mux 0 In 4
ap6 Mux 0 In 5
ap7 Mux 0 In 6
ap8 Mux 0 In 7
ap9 Mux 0 In 8
ap10 Mux 0 In 9
ap11 Mux 0 In 10
ap12 Mux 0 In 11, Out
ap13 Mux 0 In 12
ap14 Mux 0 In 13
ap15 Mux 0 In 14
ap16 Mux 0 In 15
ap17 Mux 0 In 16
ap18 Mux 0 In 17
ap19 Mux 0 In 18
ap20 Mux 0 In 19
ap21 Mux 0 In 20
ap22 Mux 0 In 21
ap23 Mux 0 In 22
ap24 Mux 0 In 23
ap25 Qstats Source
ap26 RCVT 0 1G I/O
ap27 RCVT 1 1G I/O
ap28 RCVT 2 1G I/O
ap29 RCVT 0 10G I/O
ap30 RCVT 1 10G I/O
ap31 RCVT 2 10G I/O
ap32
ap33 Mux 0 In 24
ap34 Mux 0 In 25
ap35 Mux 0 In 26
ap36 Mux 0 In 27
ap37 Mux 0 In 28
ap38 Mux 0 In 29
ap39 Mux 0 In 30
ap40 Mux 0 In 31
ap41 Mux 0 In 32
ap42 Mux 0 In 33
ap43 Mux 0 In 34
ap44 Mux 0 In 35
ap45 Mux 0 In 36
ap46 Mux 0 In 37
ap47 Mux 0 In 38
ap48 Mux 0 In 39
ap49 Mux 0 In 40
ap50 Mux 0 In 41
ap51 Mux 0 In 42
ap52 Mux 0 In 43
ap53 Mux 0 In 44
ap54 Mux 0 In 45
ap55 Mux 0 In 46
ap56 Mux 0 In 47
ap57 Mux 1 In 0
ap58 Mux 1 In 1
ap59 Mux 1 In 2
ap60 Mux 1 In 3, Out
ap61 Mux 1 In 4
ap62 Mux 1 In 5
ap63 Mux 1 In 6
ap64 Mux 1 In 7
ap65 Mux 1 In 8
ap66 Mux 1 In 9
ap67 Mux 1 In 10
ap68 Mux 1 In 11
The show fpga mode command can be used to determine what mode the application is currently set to. After
the above configuration, the show fpga mode command will work as follows:
hostname(config-app-metamux)> show fpga mode
FPGA Mode Description
-------------------- ------------ ----------------------------------
mezzanine.central 1x48 One 48-to-1 multiplexer
mezzanine.leaf_a 1x12 One 12-to-1 multiplexer
mezzanine.leaf_b
The following figures are examples of how modes within MetaMux may be arranged:
Mode Figure
1x32 Mode 1x32.
4x8 Mode 4x8.
A very common use case for MetaMux is as an exchange facing switch. This provides much lower latency
than traditional solutions, but, more importantly, provides a fixed upper bound on the latency in the absence of
contention for the outbound port (within 10 ns of the average latency).
Consider the case where there are seven downstream ports and one upstream port. Front panel port 1 is wired
to an exchange cross connect patch panel. Front panel port 2 is connected to an existing Layer 3 switch (which
continues to maintain BGP, PIM, etc). Front panel ports 3 through 8 are connected to trading servers which require
low latency access to the exchange.
Packets from the servers and the switch should be multiplexed into the exchange cross connect, and the exchange
cross connect downstream feed should be replicated to the switch and all of the servers.
Here we discuss two components of the configuration – upstream and downstream. In combination, these will
allow the above wiring scheme to operate.
These configuration commands will cause each of the downstream connections to transmit what is being received
from the exchange:
hostname> en
hostname> conf
hostname(config)> int et2-8
hostname(config-if-et2-8)> source et1
That is, front panel ports 2 through to 8 will transmit what is being received on port 1. This is done in the matrix
switch, and so there is very little latency (around 4 ns), and almost no variation in that latency. This diagram shows
what is going on conceptually:
To configure the up-stream component using the Mux application, the mux application must first be enabled, and
then the apX ports should be connected to the appropriate front panel sources using the source command.
5.5.1 Counters
The MetaMux 3.x modes incorporate packet counters on each multiplexer input and output port. The counters
include the number of packets received and transmited plus the packets that are dropped. Additionally, the number
of packet errors detected are reported. Counters are displayed using the command show mux counters when
in the config-app-metamux CLI mode or with show metamux counters when in privileged mode.
The following commands display the counter values in a scenario where the MetaMux is configured in mode 6x8
on the main FPGA where all the mux inputs are being driven by only one source of packets at full 10G rate where
all the counters can be seen in action:
hostname> en
hostname> conf
hostname(config)> app metamux
RxPackets is the total number of packets that have been received on the corresponding input.
RxDrops is the total number of packets dropped due to the buffer overflowing.
RxErrors is the total number of errored packets detected on the corresponding input.
TxPackets is the total number of packets that have been transmitted on the corresponding output.
Queuing statistics (Qstats) of packets entering the input queues of every MUX are collected every second by the
FPGA. These statistics are transmitted to a 1G output application port (AP). This output AP is connected to a 1G
management interface that is being monitored by the MetaMux application.
6.1 Configuration
The configuration of the Qstats system is done at application startup. This means that any changes to the configu-
ration will require a reset of the Qstats system. This is done by restarting the MetaMux application.
The management interface that is used by the Qstats system can be changed with the command qstats
interface available in metamux mode. Further configuration of the local network between the mentioned
management interface and the FPGA Qstats output port can be changed with the qstats interface ip
command in case it needs to be changed by the user. The current management interface and the Qstats interface IP
can be checked with the show qstats interface and show qstats interface ip CLI commands
available in MetaMux mode.
Check the corresponding commands definition in this guide for more details.
6.2 Statistics
The application collects all the Qstats that the FPGA transmits and makes them available through the show
metamux qstats command:
hostname(config)> show metamux qstats
Port Mux Average(ns) NumberOfPkts MaxQueueTime(ns) Error
---- ------ ----------- ------------ ---------------- -----
ap1 MUX 0 0 0 0 0
ap2 MUX 0 0 0 0 0
ap3 MUX 0 0 0 0 0
ap4 MUX 0 0 0 0 0
ap5 MUX 0 0 0 0 0
ap6 MUX 0 0 0 0 0
ap7 MUX 0 0 0 0 0
ap8 MUX 0 0 0 0 0
ap9 MUX 1 0 0 0 0
ap10 MUX 1 0 0 0 0
ap11 MUX 1 0 0 0 0
ap12 MUX 1 0 0 0 0
ap13 MUX 1 0 0 0 0
ap14 MUX 1 0 0 0 0
33
MetaMux App User Guide, Release 3.6.0
ap15 MUX 1 0 0 0 0
ap16 MUX 1 0 0 0 0
ap17 MUX 2 0 0 0 0
ap18 MUX 2 0 0 0 0
ap19 MUX 2 0 0 0 0
ap20 MUX 2 0 0 0 0
ap21 MUX 2 0 0 0 0
ap22 MUX 2 0 0 0 0
ap23 MUX 2 0 0 0 0
ap24 MUX 2 0 0 0 0
ap33 MUX 3 0 0 0 0
ap34 MUX 3 0 0 0 0
ap35 MUX 3 0 0 0 0
ap36 MUX 3 0 0 0 0
ap37 MUX 3 0 0 0 0
ap38 MUX 3 0 0 0 0
ap39 MUX 3 0 0 0 0
ap40 MUX 3 0 0 0 0
ap41 MUX 4 0 0 0 0
ap42 MUX 4 0 0 0 0
ap43 MUX 4 0 0 0 0
ap44 MUX 4 0 0 0 0
ap45 MUX 4 0 0 0 0
ap46 MUX 4 0 0 0 0
ap47 MUX 4 0 0 0 0
ap48 MUX 4 0 0 0 0
ap49 MUX 5 0 0 0 0
ap50 MUX 5 0 0 0 0
ap51 MUX 5 0 0 0 0
ap52 MUX 5 0 0 0 0
ap53 MUX 5 0 0 0 0
ap54 MUX 5 0 0 0 0
ap55 MUX 5 0 0 0 0
ap56 MUX 5 0 0 0 0
These statistics are stored in the local Influx database under the metamux_qstats measurement. Check the
MOS documentation for details on how to read the telemetry from Influx.
6.2. Statistics 34
CHAPTER 7
Telemetry
As default, when the Metamux application is started a background process called metamuxd runs. This process
periodically collects the statistics available in the show metamux qstats command and show metamux
counters command. Then, it stores them in the local time-series influxdb database.
7.1 Configuration
These statistics can then be easily accessed remotely using the telemetry viewer interface or forwarded to
another remote influxdb server (see MOS user manual).
The following command is to enable the telemetry viewer(Chronograf):
hostname(config)#management telemetry
hostname(config-mgmt-telemetry)#telemetry viewer
7.2 Statistics
This section shows examples of metamux telemetry. Enter the following link in the browser:
https://hostname/telemetry
Once Chronograf has been launched, click the Explore tab and choose telegraf.mm.
35
MetaMux App User Guide, Release 3.6.0
7.2. Statistics 36
CHAPTER 8
The primary use case of RCVT is to allow users to connect 10G ports to the management platform of devices with
only 1G management ports.
Of the supported MetaMux device families, 7130E-Series is the only one without 10G management ports. For
this reason, RCVT is only available on E-Series devices.
8.2 Overview
The MetaMux application provides three Ethernet Rate Converters (RCVT). Each RCVT uses two Application
Ports (APs); a 1G port and a 10G port. The 10G port is used as the input of the 10G -> 1G path, and the ouput of
the 1G -> 10G path. The 1G port is used as the input of the 1G -> 10G path, and the output of the 10G -> 1G path.
RCVT port mappings vary from one device type to another. The show metamux rcvt counters CLI command
displays all available RCVT ports in the column labelled Path, along with the Ethernet rate of the port, and what
RCVT it belongs to (RcvtId).:
hostname(config)> show metamux rcvt counters
RcvtId Path RxPackets RxDrops RxOverMTU
------ --------------------- --------- ------- ---------
0 ap29(10G) -> ap26(1G) 0 0 0
1 ap30(10G) -> ap27(1G) 0 0 0
2 ap31(10G) -> ap28(1G) 0 0 0
0 ap26(1G) -> ap29(10G) 0 0 0
1 ap27(1G) -> ap30(10G) 0 0 0
2 ap28(1G) -> ap31(10G) 0 0 0
37
MetaMux App User Guide, Release 3.6.0
ET ports are set to 10G rate by default. Since the above example is a 10G -> 1G path, the output port’s rate needed
to be changed to 1G. Giving the show metamux rcvt counters a valid RcvtId will only display the counters of
the specified RCVT.:
hostname(config-if-et1)> show metamux rcvt counters 0
RcvtId Path RxPackets RxDrops RxOverMTU
------ --------------------- --------- ------- ---------
0 ap29(10G) -> ap26(1G) 100 0 0
0 ap26(1G) -> ap29(10G) 0 0 0
The RxPackets column displays the number of packets received by the 10G and 1G RCVT ports.
Packets received by ap26 will be transmitted out of ap29.:
hostname(config)> int ap26
hostname(config-if-ap26)> source et3
hostname(config-if-ap26)> int et4
hostname(config-if-et4)> source ap29
hostname(config-if-et4)> int et3
hostname(config-if-et3)> speed 1g
hostname(config-if-et3)> source mac
hostname(config-if-et3)> loopback
hostname(config-if-et3)> pbert gen 100
hostname(config-if-et3)> show int et3-4 counters
Port RxOctets RxUcastPkts RxMcastPkts RxBcastPkts
---- -------- ----------- ----------- -----------
et3 11,350 100 0 0
et4 0 0 0 0
Since the above example is a 1G -> 10G path, the input port’s rate needed to be changed to 1G.
10G -> 1G rate conversion requires received packets to be buffered. It is possible to exceed the buffer’s limit by
sending a large burst of packets to the 1G port.:
hostname(config-if-et3)> int et1
hostname(config-if-et1)> clear counters
hostname(config-if-et1)> pbert gen 10000
hostname(config-if-et1)> show int et1-2 counters
Port RxOctets RxUcastPkts RxMcastPkts RxBcastPkts
---- --------- ----------- ----------- -----------
et1 7,779,349 10,000 0 0
et2 0 0 0 0
8.2. Overview 38
MetaMux App User Guide, Release 3.6.0
In the above example, 3856 packets were dropped. The RxDrops column in show metamux rcvt counters dis-
plays the number of dropped packets due to buffer overflow.:
hostname(config-if-et1)> show metamux rcvt counters 0
RcvtId Path RxPackets RxDrops RxOverMTU
------ --------------------- --------- ------- ---------
0 ap29(10G) -> ap26(1G) 10100 3856 0
0 ap26(1G) -> ap29(10G) 100 0 0
The Maximum Transmission unit (MTU) allowed by RCVT is 1600Bytes. The number of received packets that
are larger than MTU are displayed in the RxOverMTU column of show metamux rcvt counters.:
hostname(config-if-et1)> pbert gen 1 2000
hostname(config-if-et1)> show metamux rcvt counters 0
RcvtId Path RxPackets RxDrops RxOverMTU
------ --------------------- --------- ------- ---------
0 ap29(10G) -> ap26(1G) 10101 3856 1
0 ap26(1G) -> ap29(10G) 100 0 0
8.2. Overview 39
CHAPTER 9
This custom mode provides flow control back pressure for each ingress port in order to prevent packet loss (drops).
Each ingress ap port independently issues IEEE Pause Frames (with multicast destination MAC address) when
the ingress buffer reaches the high threshold. It then stops sending when the buffer falls below the low threshold.
This mode is only available on the 7130E & EP platforms.
If the source device does not respond to the Pause Frames (or not in time), the buffer will fill to the buffer high
threshold and start dropping packets accordingly.
The return path from the destination device (e.g. an exchange server connected to the mux output) is completely
independent of the Pause Frame return path. That is, destination return packets are NOT aggregated with Pause
Frames.
Note: This mode DOES NOT have the 10/1Gbps rate conversion ports offered in the standard E-Series modes
Note: This mode has a higher latency than the standard E-Series modes. Please refer to the latency summary
tables in the Overview section.
9.1 Configuration
A number of flow control settings are user configurable to help customise to specific applications. To show the
configuration run the following command:
hostname(config-app-metamux)> show pause frame config
High Threshold Low Threshold Source MAC Interval
-------------- -------------- ----------------- --------
50% 40% 7c:53:4a:ac:19:a9 115900ns
Sets the buffer threshold that when hit, starts the issue of IEEE Pause Frames. (Percentage):
hostname(config-app-metamux)> pause frame high threshold 50
Sets the buffer hysteresis threshold that when hit, stops the issue of IEEE Pause Frames (Percentage):
40
MetaMux App User Guide, Release 3.6.0
9.1.3 Interval
Sets the interval that the 64byte IEEE Pause Frames are issued (ns):
hostname(config-app-metamux)> pause frame interval 35500
Sets the source MAC address used for the IEEE Pause Frames:
hostname(config-app-metamux)> pause frame srcmac AA:BB:CC:DD:EE:FF
9.1. Configuration 41
CHAPTER 10
Each MUX ingress port has a rate limit module which throttles the packet rate upon exceeding a configurable
threshold.
The threshold is measured in number of packets per given time interval.
Seperate thresholds can be set for the two packet type (i.e. multicast and unicast).
10.1 Configuration
Use the following commands to configure the packet rate thresholds (number of packets per time interval):
• Set threshold for any types of packets:
pkt-per-interval storm-control any mux N input M NUMBER
• Set time interval (seconds):
interval storm-control mux N input M FLOAT
N is the range of MUX ids to apply the threshold to. M is the range of ingress ports to apply the threshold to.
The range of possible values for each threshold is 0x0 to 0xFFFFFFFE. The value 0xFFFFFFFF disables rate
limiting. By default, rate limiting is disabled and the time interval is set to one second.
If the user expects to have separate thresholds for multicast and unicast packets, they can use the following com-
mands:
• Set threshold for multicast packets:
pkt-per-interval storm-control multicast mux N input M NUMBER
• Set threshold for unicast packets:
pkt-per-interval storm-control unicast mux N input M NUMBER
Once either multicast or unicast thresholds are set, the any threshold will be disabled, and vice versa.
For low latency reasons, once any of the configured thresholds are exceeded, all ingress packets will be dropped
(regardless of type) until the rate is back below the threshold. Alternatively, the rate limit module can be configured
to enter the Storm Killed state upon exceeding any of the configured thresholds.
In this state, all incoming packets are dropped until the user manually restarts the link.
The user can enable this feature with the command:
overrate-kill storm-control mux N input M
The user can recover a client port in the “Storm Killed” state with the command:
42
MetaMux App User Guide, Release 3.6.0
A counter of packets that are dropped due to exceeding the Storm threshold is pushed to telemetry under the key
storm_drop_cnt. It is also in the output of:
show metamux counters verbose
10.2 Example
In the following example, we will confgure the Packet Per Interval (PPI) threshold of Input 0 in MUX 0 to 10 any
packets per second. This means that Input 0 will drop packets to keep the rate below 10 packets per second:
hostname(config)# app metamux
hostname(metamux)# pkt-per-interval storm-control any mux 0 input 0 10
Next, we will configure the PPI threshold of Input 1 in the same mux to 10 multicast packets and 5 unicast packets
per interval, as well as change the Interval to half a second. This means that Input 1 will drop packets to keep the
rate below 20 multicast packets per second (or 10 packets per half second) and 10 unicast packets per second:
hostname(metamux)# pkt-per-interval storm-control multicast mux 0 input 1 10
hostname(metamux)# pkt-per-interval storm-control unicast mux 0 input 1 5
hostname(metamux)# interval storm-control mux 0 input 1 0.5
Note: Input 1 will drop any type of packet to try and keep the rate down, it does not filter based on packet type.
Finally, we will configure the threshold of Input 2 in the same mux to 10 unicast packets, as well as turn on
overrate-kill for that port. This means that Input 2 will now drop all packets after it detects a rate of 10 unicast
packets per second. Input 2 will not allow anymore packets through until overrate-kill is either turned off or
restarted:
hostname(metamux)# pkt-per-interval storm-control unicast mux 0 input 2 10
hostname(metamux)# overrate-kill storm-control mux 0 input 2
If a continuous stream of multicast packets that exceed 20 packets per second was sent to all 3 ports, the
StormKilled status of the 3 ports we configured should look like:
hostname(metamux)# show metamux ap1-3 status
Port Name Source Locked Enabled RxPeriod StormKilled
----- ----------- -------- -------- -------- ------------ ------------
ap1 Mux 0 In 0 et1 Y Y 0x001FFFF7 Y
10.2. Example 43
MetaMux App User Guide, Release 3.6.0
In the above case, only Input 0 and Input 1 will start dropping packets. Looking at MetaMux’s verbose counts, we
can see that packets will still flow through the ports, but only at the rate that was configured:
hostname(metamux)# show metamux ap1-3 counters verbose
Port Name RxPackets RxBuffered RxDrops RxErrors RxMTUErr StormDrop
---- ----------- ---------- ----------- -------- --------- --------- ----------
ap1 Mux 0 In 0 70 60 0 0 0 2160457
ap2 Mux 0 In 1 140 127 0 0 0 2160317
ap3 Mux 0 In 2 2160527 1660052 392472 0 0 0
If we were to clear the MetaMux counters and change the continuous stream to a burst of 1000 unicast packets,
the above commands will yield the following:
hostname(metamux)# show metamux ap1-3 status
Port Name Source Locked Enabled RxPeriod StormKilled
----- ----------- -------- -------- -------- ------------ ------------
ap1 Mux 0 In 0 et1 Y Y 0x001FFFF7 N
ap2 Mux 0 In 1 et1 Y Y 0x001FFFF7 N
ap3 Mux 0 In 2 et1 Y Y 0x001FFFF7 Y
hostname(metamux)# show metamux ap1-3 counters verbose
Port Name RxPackets RxBuffered RxDrops RxErrors RxMTUErr StormDrop
---- ----------- ---------- ----------- -------- --------- --------- ----------
ap1 Mux 0 In 0 10 0 0 0 0 990
ap2 Mux 0 In 1 10 10 0 0 0 990
ap3 Mux 0 In 2 10 10 0 0 0 990
All ports let only 10 packets through because of their any and unicast PPI configurations. However, only Input 2
keeps its StormKilled status because overrate-kill is turned on for that port. If we send 1 packet to all the ports,
the MetaMux counters will look like:
hostname(metamux)# show metamux ap1-3 counters verbose
Port Name RxPackets RxBuffered RxDrops RxErrors RxMTUErr StormDrop
---- ----------- ---------- ----------- -------- --------- --------- ----------
ap1 Mux 0 In 0 11 0 0 0 0 990
ap2 Mux 0 In 1 11 11 0 0 0 990
ap3 Mux 0 In 2 10 10 0 0 0 991
Input 2 is the only port that dropped the packet. The following will restart overrate-kill for Input 2:
hostname(metamux)# overrate-kill storm-control mux 0 input 2 restart
hostname(metamux)# show metamux ap1-3 status
Port Name Source Locked Enabled RxPeriod StormKilled
----- ----------- -------- -------- -------- ------------ ------------
ap1 Mux 0 In 0 et1 Y Y 0x001FFFF7 N
ap2 Mux 0 In 1 et1 Y Y 0x001FFFF7 N
ap3 Mux 0 In 2 et1 Y Y 0x001FFFF7 N
This will now allow packets to pass through Input 2. Finally, the following commands will clear all the configura-
tions we set:
no pkt-per-interval storm-control any mux 0 input 0
no pkt-per-interval storm-control multicast mux 0 input 1
no pkt-per-interval storm-control unicast mux 0 input 1
no pkt-per-interval storm-control unicast mux 0 input 2
no interval storm-control mux 0 input 1
no overrate-kill storm-control mux 0 input 2
10.2. Example 44
CHAPTER 11
The Elastic Buffer modes are provided for customers that need to relax the ingress port clocking requirements
needed for the ultra low latency implementations of MetaMux. This occasionally occurs for older devices where
the “clock wander” is higher than tolerated by MetaMux, causing buffer over/underrun internally (packet corrup-
tion).
Additionally, the Elastic Buffer modes are required if a customer arranges MetaMux instances into a Clocking
Feedback Loop. A Clocking Feedback Loop can only occur between instances of MetaMux running on separate
devices. The image bellow illustrates an example of a Clocking Feedback Loop.
Device A and Device B are two devices running their own instances of MetaMux. What makes it a Clocking
Feedback Loop is an output of a Multiplexer in MetaMux Device A is being sourced by a Multiplexer’s input in
MetaMux Device B, and an output of a Multiplexer in MetaMux Device B is being sourced by a Multiplexer input
in MetaMux Device A. The software of the MetaMux instances in Device A and Device B will adjust their output
clocks by tracking the frequencies of their input clocks. This means that an adjustment in Device A’s output clock
will lead to an adjustment in Device B’s output clock, which will then lead to another adjustment in Device A’s
output clock, etc...
If the same topology were to be used in a single instance of MetaMux, the software can detect and break the loop.
However, in the example above each instance of MetaMux is not aware of what the other instance is doing. Using
an Elastic Buffer mode in any of the MetaMux instances will break the feedback loop because the software does
not need to adjust the the Multiplexer’s output clock in the Elastic Buffer mode.
The following are the current available modes:
45
MetaMux App User Guide, Release 3.6.0
Mode Description
------------ ------------------------------------------------------------
6x8_eb Six 8-to-1 multiplexers with elastic buffers
2x8+2x16_eb Two 8-to-1 and two 16-to-1 multiplexers with elastic buffers
Elastic Buffer modes have a higher measured latency than the normal MetaMux Modes do. Please refer to the
Latency section to compare the Latency results of the Elastic buffer modes and the normal modes.
46
CHAPTER 12
Command Reference
47
MetaMux App User Guide, Release 3.6.0
no shutdown (config-app-metamux)
Enable the MetaMux application.
shutdown (config-app-metamux)
Disable the MetaMux application. This is the default state.
mode (config-app-metamux)
set mux mode
mode NAME
NAME
A string
Note that if the application is running (no shutdown) the FPGA will be re-programmed.
Example:
hostname(config-app-metamux)#mode 1x48
hostname(config-app-metamux)#
no mode (config-app-metamux)
remove metamux mode
no mode
This command removes the metamux mode, it fails if the application is running.
default mode (config-app-metamux)
set metamux to default mode (no mode)
default mode
Note that if the application is currently running it will be shutdown first. This command is equivalent to:
hostname(config-app-metamux)# shutdown
hostname(config-app-metamux)# no mode
Appendix
13.1.1 C96E-series
58
MetaMux App User Guide, Release 3.6.0
13.1.2 E-series
13.1.3 EB-series
13.1.4 L-series
13.1.5 LB-series
113
MetaMux App User Guide, Release 3.6.0
Q
qstats interface (config-app-metamux), 54
qstats interface ip (config-app-metamux), 54
R
restart daemons (config-app-metamux), 57
S
scheduler (config-app-metamux), 51
show fpga mode (config-app-metamux), 49
show loglevel (config-app-metamux), 50
show metamux mode (priv), 47
show metamux mode structure all (priv), 47
show metamux rcvt counters [NUMBER] (priv), 47
show metamux status (priv), 48
show metamux [INTERFACE] counters [verbose]
(priv), 47
show metamux [INTERFACE] qstats (priv), 48
show mode (config-app-metamux), 49
show mode structure all (config-app-metamux), 49
show mux status (config-app-metamux), 50
show mux [INTERFACE] counters [verbose] (config-
app-metamux), 50
show mux [INTERFACE] qstats (config-app-
metamux), 54
show pause frame config (config-app-metamux), 55
show qstats interface (config-app-metamux), 54
show qstats interface ip (config-app-metamux), 54
show qstats source (config-app-metamux), 55
shutdown (config-app-metamux), 48
start qstats (config-app-metamux), 57
stop qstats (config-app-metamux), 57
support information, 2
Index 114