TMV Basic Testing Rev005 17112021 - Clean
TMV Basic Testing Rev005 17112021 - Clean
White Paper
DOI: 10.5281/zenodo.5704519
URL: https://doi.org/10.5281/zenodo.5704519
Executive Summary
This document (Basic Testing Guide) is a practical guide describing the starter kit developed in the context
of the 5G PPP Test, Measurement and KPI Validation work group. The guide enables the interested
developer to understand how this can be applied to measure and verify basic 5G KPIs. The document
starts from describing the idea, the intention to measure, up to the actual realization of the test. To enable
the test, a description of the environment, how to install it, the test tools and the methodology is
provided. It guides the developer, step by step, to run the intended test. The proposed framework with
the support of the described android agents can in principle support the validation of application level
KPIs running at the android UEs, while for other application-specific KPIs running at other systems, the
development of specialized plugins is necessary.
2|Page
5G PPP TMV WG White Paper: Basic Testing Guide
The Group is comprised by several Phase II and Phase III 5G PPP projects, and it considers the following research
areas and technology domains:
• Testing KPI definition, KPI sources, collection procedures and analysis
• Testing frameworks (requirements, environment, scenarios, expectations, limitation) and tools
• Testing methodologies and procedures
• KPI validation methodologies
• Testing lifecycle (i.e. testing execution, monitoring, evaluation and reporting)
• Common information models for 5G T&M
Another important topic is the use of and contribution towards open-source projects such as OSM, OPNFV
or ONAP and identification of relevant exploitation and dissemination targets to promote the European
vision on T&M towards a more global adoption.
Motivations
It is often the case that there exists a gap between theory and application which also is the case when it
comes to KPI verification. This comes to show when studying material on KPI definitions and formalized
test cases and considering how to apply this in a specific scenario. This process might cover tools selection,
deployment and configuration, scripting and automation, test execution and result extraction. These steps
might seem like a high entry barrier and require significant resources both in selecting and applying the
approach. This means that there are many things that need to be in place even before starting to execute
test cases.
This is the background for creating this basic testing bundle which will allow the test engineer to hit the
ground running when starting to do KPI verification. The bundle provides the test engineer with an initial
setup of tools and means to automate the control of the tools.
The main focus is to provide the foundation for automation of tests, which is critical when wanting to
expand the basic tests with more advanced tests. This is also highly useful if integrating the testing as part
of CI/CD pipelines for developing and deploying 5G VNFs or services.
3|Page
5G PPP TMV WG White Paper: Basic Testing Guide
similar documents coming from NGMN [1] and 3GPP [2]. The former is focused on testing aspects, while
the latter provides an overview of KPIs to be measured during normal network operations.
Testing is normally applied during the network rollout, during the onboarding process of new Virtualized
Network Functions (VNFs) or new software updates, and for the validation of newly deployed network
services. Monitoring is instead constantly active during network operations and it is a key enabler of
network management actions and processes.
• P1: Providing 1000 times higher wireless area capacity and more varied service capabilities
compared to 2010.
• P4: Creating a secure, reliable and dependable Internet with a “zero perceived” downtime for
services provision.
In practice, those two contractual KPIs can be translated in a series of Technical KPIs, as displayed in Table
1.
Table 1 - Identified Technical KPIs
5G PPP KPI
Type KPI name KPI measurement points
Validated
The definition of these KPIs has mostly been derived or reworked starting from [7] and [8]. The “Type”
column remarks for which purpose the KPI is useful. The SLA KPIs are the ones used to validate if the
service design can support the SLA agreed with the vertical, and they can used as well during the network
4|Page
5G PPP TMV WG White Paper: Basic Testing Guide
monitoring phases to trigger alarm and network management actions. The Technology Validation KPIs are
instead focused on providing the proof that 5G is delivering the promised performances, mostly peak
ones.
With the test case defined formally it can now be translated into a test plan consisting of several steps
that define the inputs, perform the actions, apply the stimuli, extract outputs, and evaluate the results.
The test plan should be reproducible and should be straight forward to follow to perform the test
actions. The sequence of steps in the test plan is what will be automatized such that it can easily be re-
executed in the same context or in another context.
This translation is where OpenTAP comes into play and excels. A test plan in OpenTAP is comprised of a
sequence of test steps, which each perform a specific action, often interacting with a specific tool.
Furthermore, the resources to be used by the test plan are defined externally to the test plan as
instruments and DUTs. This makes it easy to define new resources and apply the same test plan in a
different context.
5|Page
5G PPP TMV WG White Paper: Basic Testing Guide
measurement instruments and possibly vendor-specific Devices Under Test (DUTs). By leveraging the
features of C#/.NETcore and providing an extendable architecture, OpenTAP minimizes the code needed
to be written by the programmer. OpenTAP is distributed under the Mozilla Public License (MPL).
OpenTAP offers a range of functionality and infrastructure for configuring, controlling and executing test
algorithms. OpenTAP provides an API for implementing plugins in the form of test steps, instruments,
DUTs and more.
OpenTAP consists of multiple executables, including:
• OpenTAP (as a dll)
• Command Line Interface (CLI)
• Package Manager
Steps frequently have dependencies on DUT and instrument plugins.
The illustration below shows how OpenTAP is central to the architecture, and how plugins (all the
surrounding items) integrate with it.
If a graphical user interface is needed you can download the Keysight Test Automation Developer's System
(Community or Enterprise Edition). It provides you with both a Software Development Kit (SDK) as well as
an Editor GUI. Please check the EULA is the use of Community Edition GUI applies to you.
An essential feature of OpenTAP is its flexible architecture that lets users create plugins. OpenTAP plugins
can be any combination of Test Step, Instrument, and DUT implementations. Other OpenTAP components
such as Result Listeners and Component Settings are also plugins. By default, OpenTAP comes with plugins
covering basic operations, such as flow control and result listeners.
Plugins are managed by the Plugin Manager, which by default searches for assemblies in the same
directory as the running executable (the GUI, CLI or API). Additional directories to be searched can be
specified for the GUI, the CLI and the API.
6|Page
5G PPP TMV WG White Paper: Basic Testing Guide
A test plan is a sequence of Test Steps with some additional data attached. Test plans are created via the
Editor GUI. Creating test plans is described in the Graphical User Interface Help, accessible within the
Editor GUI. Test plan files have the .TapPlan suffix, and are stored as xml files.
TapPlan Settings
Editable OpenTAP step settings can be marked as TapPlan Settings. The value of such settings can be set
through the Editor GUI, through an external program (such as OpenTAP CLI), or with an external file. This
gives the user the ability to set key parameters at run time, and (potentially) from outside the Editor GUI.
OpenTAP Documentation
OpenTAP documentation is provided from the OpenTAP project [9]. Here is also provided developer
guides and information about how to join and contribute to the community. Here can also be found links
for the publicly available OpenTAP Package repository from where plugins can be downloaded.
7|Page
5G PPP TMV WG White Paper: Basic Testing Guide
Testing tools
Below are short descriptions of the tools that can be controlled by the OpenTAP plugins pre-installed in
the Basic Testing bundle.
IPerf
iPerf3 is a tool for active measurements of the maximum achievable bandwidth on IP networks. It
supports tuning of various parameters related to timing, buffers and protocols (TCP, UDP, SCTP with
IPv4 and IPv6). For each test it reports the bandwidth, loss, and other parameters. This is a new
implementation that shares no code with the original iPerf and is not backwards compatible.
iPerf features
Yardstick
Yardstick is an open-source project developed by the OPNFV community. The goal of the Yardstick Project
is to verify the infrastructure compliance when running VNF applications.
NFV Use Cases described in ETSI GS NFV 001 show a large variety of applications, each defining specific
requirements and complex configuration on the underlying infrastructure and test tools. The Yardstick
concept decomposes typical VNF work-load performance metrics into several
characteristics/performance vectors, which each of them can be represented by distinct test-cases.
The project's scope is to develop a test framework, test cases and test stimuli.
Yardstick allows to:
• Decompose VNF work-load performance metrics into a number of characteristics / performance
vectors, identifying and categorizing the metrics related to characterization of the
infrastructure, develop test case examples to realize the metrics;
8|Page
5G PPP TMV WG White Paper: Basic Testing Guide
• Enable verification of more complex test cases by developing functionality to run: parallel
testing, inject fault, test multiple topologies, test scenarios.
The methodology used by the Project, to verify the infrastructure form the perspective of a VNF, is fully
aligned with ETSI NFV-TST 001. Visualization of the test results is performed using Grafana as shown
below:
Android Agents
The Android Agents plugin contains steps and instruments for controlling four different agents that can
be installed in Android devices. Additionally, the Android plugin, which is a requirement for the use of
the plugin provides several generic steps for using android devices through ADB (Android Debug Bridge).
The following agents are included:
- iPerf Agent: The iPerf agent is an Android application that acts as a wrapper for the iPerf
measuring tool, which is bundled with the provided apk, and is able to control two separate
instances at the same time, one acting as client and the other acting as server. The included test
9|Page
5G PPP TMV WG White Paper: Basic Testing Guide
steps allow the configuration of both instances separately, as well as the retrieval and publishing
of the generated results.
- Ping Agent: Likewise, this agent also acts as a wrapper to the native ping utility included in
Android devices, automatically generating results based on the output of the command.
- Resource Agent: The resource agent allows the monitoring of the radio parameters along with
the usage of the device resources.
10 | P a g e
5G PPP TMV WG White Paper: Basic Testing Guide
- Exoplayer Agent: Exoplayer is an alternative MediaPlayer API for Android devices, with support
for DASH and SmoothStreaming adaptive playbacks. The Android Agents plugin provides a
customized version of the demo application of the ExoPlayer library that is prepared for the
generation of measurements. The included test step is able to initiate and stop the playback of a
video using the application, as well as publishing the resources obtained.
The OpenTAP container comes with a number of open-source plugins installed (also from the 5GENESIS
project github.com/5genesis), which can be used to control and interact with tools or entities, but also
with a TapPlans editor (TUI) to create and modify test plans.
The Tools container comes with ssh, iperf3 and ping-utils installed.
Deployment of the containers can be done using any container runtime, but as an example the project
provides a docker-compose file. This file defines an example of how docker-compose can be used to
deploy a simple infrastructure with 3 containers - 1 OpenTAP and 2 tools.
11 | P a g e
5G PPP TMV WG White Paper: Basic Testing Guide
The same approach can be used can be used to install additional OpenTAP plugins from OpenTAP
repositories. Similarly packages can also be installed based on local .TapPackage files.
The example TapPlan performs an Iperf upload and download test, prints the unix name and performs a
ping between the 2 deployed tools containers. The Iperf OpenTAP plugin allows for basic configuration
of TCP/UDP uplink and downlink throughput tests initiated from an Iperf3 client and server.
The TestPlan is structured as a set of nested test steps, with each one configuring one of the different
agents. The duration of the TestPlan is controlled by the last one (Adb Exoplayer), while the others are
run for the duration of the child steps.
In order to be able to run the TestPlan, an Android device must be connected to the test machine via
USB, the Agents must be installed in the device, and the corresponding Instruments must be configured
12 | P a g e
5G PPP TMV WG White Paper: Basic Testing Guide
in OpenTAP. Also note that the Adb iPerf Agent step must be configured with the Host and Port values
of a running iPerf 2 server in order to generate any results.
Conclusions
By reading this guide, the authors hope you are now able to create and run useful tests to validate your
5G KPIs. The authors welcome contribution or package expansion proposed via Gitlab. If you have any
issue, please post your comments in the Gitlab repository.
Contributing Projects
• 5G-VINNI
• 5GENESIS
Contacts
WG Chair: Evangelos Kosmatos, WINGS ICT Solutions [email protected]
Contributors
Name Company/Institution/University Country
Editors
Andrea F. Cattoni Keysight Technologies, Denmark
Lars Nielsen Keysight Technologies Denmark
Contributors and Reviewers
Almudena Diaz Zayas University of Malaga Spain
Bruno García García University of Malaga Spain
Lars Nielsen Keysight Technologies Denmark
Anastasius Gavras Eurescom Germany
Michael Dieudonne Keysight Technologies Belgium
References
[1] NGMN, Definition of the Testing Framework for the NGMN 5G Pre-Commercial Network Trials
(Version 2),
https://www.ngmn.org/fileadmin/ngmn/content/downloads/Technical/2019/190111_NGMN_P
reCommTrials_Framework_definition_v2_small.pdf
[2] 3GPP, TS 28.552: Management and orchestration; 5G performance measurements,
https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationI
d=3413
13 | P a g e
5G PPP TMV WG White Paper: Basic Testing Guide
[3] ETSI Zero touch network and Service Management (ZSM) Industry Specification Group (ISG),
https://www.etsi.org/technologies/zero-touch-network-service-management
[4] Open Platform for NFV (OPNFV), Linux Foundation Project, https://www.opnfv.org/
[5] Open Network Automation Platform (ONAP), Linux Foundation Project, https://www.onap.org/
[6] "Contractual Arrangement Setting up a Public Private Partnership in the Area of Advanced 5G
Network Infrastructure for te Future Internet between the European Union and the 5G
Infrastructure Association", December 17, 2013 (https://5g-ppp.eu/contract/ latest access
07/05/2019)
[7] 3GPP, TS 28.554: Telecommunication management; Management and orchestration; 5G end to
end Key Performance Indicators (KPI),
https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationI
d=3415
[8] EU H2020 TRIANGLE Project, Deliverable D2.6: Final Test Scenario and Test Specifications,
https://www.triangle-project.eu/wp-content/uploads/2018/11/TRIANGLE_D2-6.pdf
[9] OpenTAP - An Open Source Project for Test Automation https://www.opentap.io/
[10]5GPPP TMV Basic Testing Bundle Gitlab Project https://gitlab.com/OpenTAP/Plugins/5g-ppp-
test-measurement-and-kpis-validation-wg/5gppp-tmv-basic-testing-bundle
14 | P a g e