VIRTUAL

PROTECTION RELAY
A PARADIGM SHIFT IN POWER SYSTEM PROTECTION

Authors: Intel - Dean Samara-Rubio, Graham McKenzie, Prithpal Khajuria

Kalkitech - Roopesh Ariya, Prasanth Gopalakrishnan, Viju Ravindran

1
 TABLE OF CONTENTS
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 VPR Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 VPR Approach 1 – Direct Porting of discrete IEDs onto a host . . . . . . . . . . . . . . . . . . . . . . 5

2.2 VPR Approach 2 – Shared Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3 The Role of Software Container Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.4 Hardware Platform Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.4.1 Server Hardware Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.4.2 Time Coordinated Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.4.3 Precision Time Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4.4 Next Generation NIC with HSR/PRP, PTP, and SRIOV . . . . . . . . . . . . . . . . . . . . . . . 8

2.5 Hypervisor Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.5.1 VPR Redundancy and High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.5.2 Implementation of Digital twin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 VPR Reference System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1 Kalkitech VPR Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Signal Conditioning within the VPR Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.3 Laboratory Setup for Testing of VPR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.4 Evaluation of VPR Reference Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.4.1 Response Time Dependence on HW Allocation to VM . . . . . . . . . . . . . . . . . . . . .14

3.4.2 Response Time versus Fault Magnitude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 The Path Ahead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.1 Reference materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2
1. Introduction
As the energy sector transitions to a high penetration of renewable energy sources,
many of which will connect to the grid as distributed energy resources (at the medium
and low voltage levels), the patterns and levels of power flow are changing significantly.
Figure 1 illustrates the extent to which our mix of energy sources is shifting with
renewable energy production projected to exceed the sum of both Coal and Nuclear
within this decade. The power sector is responding in several ways: First, the relative
agility of natural gas plants compared to baseload power stations of coal and nuclear
is highly valued to ballast against the variability in renewable energy production and
this has accelerated the transition to natural gas (while at the same time reducing
carbon footprint compared to coal). Secondly, the adoption of bulk energy storage has
begun to further smoothen the peaks and valley of renewable energy supply. Third,
investments into increasingly adaptive “intelligent” grid controls is crucial to enable grid
operators at all levels to manage voltage levels and supply quality across the grid.

It has been well-documented that one of the key aspects of the grid controls which are
put under stress by this penetration of renewables is the circuit protection sub-system.
To operate and maintain effective circuit protection with high penetration of renewable
resources requires a level of agility and programmability for protection systems
not available in today’s mainstream protection relay products. What’s more, when
considering that there is a still a large installed base of older generation devices across
the grid, utilities are faced with a generational decision on how to re-invent and reform
their entire protection infrastructure for the agile and intelligent grid. In this whitepaper
we describe the virtual protection relay (VPR) concept – an architecture where software-
defined and virtualized platforms are deployed to host the critical circuit protection
functions for an advanced and agile grid. We assert that this use of virtualization
technologies is needed to unlock the potential of the grid to deliver reliable, available,
and resilient electricity supply for our low-carbon economy of the future.

Renewables Projected to Become Largest Electricity Source

Electricity production by source (kilowatt hours)

2.5T
Renewable Energy

2T Natural Gas

1.5T

1T

Nuclear Energy
500B Coal

0 Petroleum and Other Sources

0 5 0 5 0 5 0 5 0
201 201 202 202 203 203 204 204 205

Source: U.S. Energy Information Administration.

Figure 1: Renewables set to become the leading source of electricity. Electricity production by renewable sources is approaching an
unprecedented level, set to exceed 2T kWh per year in the USA by 2043.

3
The first protection relay was developed in 2000’s, the IEC 61850 set of standards brought
the beginning of the 1900’s beginning with in a major boost to multi-vendor IED integration
electromechanical devices that would sense and device-to-device data transfer. Standard
a fault and actuate a mechanical switch (or a models of various protection functions were
series of mechanical switches) to interrupt the devised for possible interoperability between
source that was feeding the fault with hazardous protection IEDs from any vendor. Standards were
energies. Power electronics technology later prepared for data exchange between devices
paved way to static or solid-state relays based on (station bus) and current/voltage information
electronic components in the 1960’s. Then came from field (process bus). Acceptance of the IEC
the development of Microprocessor-based relays 61850 standards worldwide have resulted in
in the 1980’s including digital signal processing station level and process level communication
and digital network communications. These networks for exchange of digitized raw values
relays along with a range of similar devices for (using Sampled Values, or SV, protocol) and
monitoring and control of the grid were given processed values/information across the
a name – “intelligent electronic device”, or IED. substation devices and beyond the substation to
While the core functionality of power system centralized monitoring systems.
protection -which is detection of a power
system disturbance and isolation of the affected Even after the century of advancement, the
area- has remained relatively unchanged, the implementation of protection for even a single
successive generations of technology have medium-voltage substation requires up to
improved the operation of the grid as it scales in 100 individual IEDs each with its own power
size and complexity. supply, case, electronic components, firmware,
installation and maintenance guide(s), and
Each step in the evolution of protection configuration tool(s). Each vendor of IEDs has
systems has been an effort to standardize the their own design paradigm for their hardware
functionality, behavior, and interfaces to enable and there is a significant engineering cost to
a robust market of vendors and solutions to swap one make or model of an IED with an
serve the global power utility sector. In the early essentially equivalent IED from another vendor.

MU IED
Control
Center
Breaker
IED
IED
Process Bus

Station Bus

IED Gateway

MU IED HMI

Breaker
IED SCADA
IED

Process Bay Station

Level Level Level

Figure 2: IEC 61850 Digital Substation Network model. Merging Unit(s) (MU), circuit breaker controllers (Breaker IED), relay and control devices (IED), and
supervisory control and data acquisition (SCADA) systems are connected across three conceptual levels with communication buses.

4
VPR eliminates many of these costs., reduces 2. VPR Architecture
the physical size and complexity of a substation
There are multiple approaches for utilizing
protection and control infrastructure, and opens
the virtualization capability to create a VPR
the market for software-defined innovation
system. Any protection system must meet
where the best-of-breed protection functions
the basic requirements of Selectivity, Speed,
and automation algorithms can be deployed
Simplicity, Reliability, Economy. In order to
with high confidence and reliability but
meet these objectives and to maintain a healthy
minimum cost.
market, additional considerations include
While these potential benefits of VPR are the maintainability by the end-customer, the
compelling, there remain questions in the supportability by the technology providers,
industry as to whether the virtualized hosts interoperability between vendor offerings,
and protection functions deployed as software and the ability for test and certification by
modules can meet the performance and independent actors.
reliability requirements of the modern grid.
In this paper, a reference VPR subsystem 2.1 VPR Approach 1 – Direct Porting of
is described, and the initial performance discrete IEDs onto a host
characteristics are presented. The VPR solution A straightforward approach will be to port
builds upon the preceding century of innovation a complete existing stand-alone IED into a
in the protection industry including the IEC- virtual machine (VM) on the host hardware and
61850 standard. hypervisor. If this is done for multiple discrete
IEDs then a system such as shown in Figure 3
would result. Note that there is no “sharing”
of functions and each of the VMs will have
its own complete ensemble of capabilities
(Sampled Value subscriber, MMS Server, grid
Figure 3: Direct Porting of Protection relay software event recording, system health monitoring,
Each Discrete IED is ported to the host in its own virtual machine.
None of the application functionality is shared. configuration utilities, etc.)

Protection Function-1 Protection Function-k Protection Function-1 Protection Function-k

Processed Data Cache Processed Data Cache

Conditioning Conditioning

Sampled Value Cache Sampled Value Cache

GOOSE - MMS Cache GOOSE - MMS Cache

9-2 SV GOOSE 9-2 SV GOOSE

Subscriber Publisher/Subscriber
MMS Server Subscriber Publisher/Subscriber
MMS Server

Process Bus Network Station Bus Network Process Bus Network Station Bus Network

Virtual IED 1 Virtual IED 2

VM 1 VM 2

5
 2.2 VPR Approach 2 – Shared Functionality Server on the Station Bus is consolidated
In an alternative approach the IED functionality and shared. With the application of advanced
is divided so that common modules are network interface virtualization (e.g., the
consolidated to a separate VM. This can give inclusion of the Single-Root Input Output
a level of efficiency when scaling to a larger Virtualization, or SR-IOV, technology), and
number of protection functions, but to do so further design optimizations and development
requires an open API to be defined so that of the common API’s, it is possible to share
each VM can access the shared functions and more of the functions including the sampled
make data available to all other modules and value subscriber, signal conditioning, system
protection functions across multiple VMs. monitoring, and other functions of the system to
For example, as shown in Figure 3, the MMS further enhance the scalability of the solution.

Protection Function-1 Protection Function-k Protection Function-1 Protection Function-k Virtual IED 1 Logical Device(s)
and Logical Node(s)

Processed Data Cache Processed Data Cache MMS Server

Virtual IED n Logical Device(s)

Conditioning Conditioning and Logical Node(s)

Sampled Value Cache Sampled Value Cache

GOOSE - MMS Cache GOOSE - MMS Cache GOOSE Cache

9-2 SV GOOSE 9-2 SV GOOSE GOOSE GOOSE

Subscriber Publisher/Subscriber
MMS Server Subscriber Publisher/Subscriber
MMS Server Subscriber Publisher

Process Bus Network Station Bus Network Process Bus Network Station Bus Network Station Bus Network

Virtual IED 1 Virtual IED 2 Virtual IED (n+1)

VM 1 VM 2 VM (n+1)

Figure 4: Porting of Relay SW with refactoring to use some shared services

Protection functions can run individually or as a group in one or more VM. Common functions like SV Subscriber, MMS Server, GOOSE publisher, configuration module,
system monitoring, etc. can be part of a different VM. To achieve this partitioning and sharing of common functions openly defined API interfaces are required.

2.3 The Role of Software Container systems. As commercial solutions for VPR
Technology emerge, we expect that both virtualization
VPR may also utilize the software container hypervisor and software containerization
technology (container systems include the well- technologies will be utilized together.
known Docker and Podman projects among
others). Container technology is a complement 2.4 Hardware Platform Technologies
to the VM hypervisor so that, for example, each 2.4.1 Server Hardware Specification
discrete protection function and supporting Today, utilities must handle multiple physical
microservice can be its own container. The IEDs from different vendors deployed during
advantage of container technology is the scaling different periods of time. They must have a
of services with maximum portability and a set survivability plan for maintenance, management,
of tools that help the software developers in and replacement, for a large ensemble of
the integration, deployment, and maintenance different product generations, product SKUs,
of their software across the lifecycle of the firmware versions, and product options. In the
6
virtualized environment all the physical IEDs internal operations as data is moved between
can be accommodated in an Intel Xeon based software processes and threads, calculations
Server hardware that can withstand the harsh are performed, etc. In the case of protection and
environment in a substation. To achieve the control functions, the timescale for deterministic
maximum effectiveness of the VPR concept, behavior at the system level is on the order of 5
Intel is working with an ecosystem of computing milliseconds.
hardware companies to design and manufacture
Intel Xeon-based Substation Servers that are In the suggested VPR environment there will be
IEC 61850-3 and IEEE 1613 certified for use in a multiple VMs hosting a heterogeneous mix of
substation and capable of hosting automation, applications. Ensuring consistent performance
control, and protection workloads. OEMs and the right priorities for access to hardware
including Advantech, Crystal Rugged, Dell and resources is a key role of the hypervisor with
others have created powerful rugged servers real-time support. Even in a model where VM’s
that meet the needs of the market. are assigned (pinned) to specific CPU cores
and memory is allocated to a VM, there can be
2.4.2 Time Coordinated Computing shared resources of the system including the
network and internal buses, and the last-level
Intel-based computing systems have cache. Intel Cache Allocation Technology (CAT)
technologies which enable the hosting time address the concerns on the most critical of the
-sensitive applications; applications with shared resource by providing software control
operations that must execute within defined of where data is allocated into the last-level
rules and time constraints. The term “real-time” cache (LLC), enabling more effective isolation
is used to identify such operations but the term and prioritization of key applications.
is used in many different contexts and not all
“real-time” systems are the same. In practice, Intel Time Coordinated Computing (Intel TCC)
response time requirements at the external including CAT enabled processors deliver
system level varies from a few microseconds optimal compute and time performance for
to hundreds of milliseconds, but for every real-time applications. Intel TCC ensure that
external action (of even an optimized software performance requirements are met in a stand-
system for “real-time” computing) there can alone environment and under the presence of
be a sequence of hundreds or thousands of concurrency with other workloads.

Board Support Package § Real-time Kernel

with real-time optimizations § Optimized drives

Firmware § Intel® TCC Mode in BIOS

with real-time optimizations § SMI reduction

§ Granular real-time tuning

Intel® TCC Tools § Real-time / Power / General
compute balance

Figure 5: Model to enable Intel Time Coordinated Computing (TCC) for real-time workloads.
7
2.4.3 Precision Time Protocol the compute platform that needs to be shared
IEDs in the substation operates in real-time between VM’s and can be one of the points
where the devices shall be time synchronized for performance degradation when hosting
to the best possible accuracy. Precision Time time-critical applications. In addition to latency
Protocol (PTP) provids time synchronization considerations, the VPR system adheres to the
for digital substations with microsecond- IEC61850 standards which dictates zero packet
level accuracy. PTP, defined in IEEE standard loss for the multiple high speed sampled value
(IEEE1588-2008), is a protocol enabling streams such that all packets should be received
precise synchronization of device clocks in in the virtualized devices.
Ethernet. Devices running PTP are automatically In the network between the powerline sensing
synchronized to the most accurate clock in the components and the VPR server exists the
network. The protocol supports system wide process bus (as defined for example in the
synchronization accuracy in sub microsecond IEC 61850-9-2 standard). The process bus is
range with minimal network and local clock typically implemented as a redundant network
computing resources. VM Hypervisor shall making use of high-availability seamless
support high accurate clock synchronization redundancy (HSR) and parallel redundancy
with dedicated ports in the network interface protocol (PRP) equipment. One approach to
card (NIC). There are network cards that VPR is to terminate the redundant network at
has HSR/PRP support along with PTP time the ingress to the VPR server on the network
synchronization feature. interface card (NIC) in hardware, thus eliminating
the software from having to process redundant
2.4.4 Next Generation NIC with HSR/PRP, packets and reduce latency. The NIC would
PTP, and SRIOV use single root I/O virtualization (SR-IOV)
The network interface is another resource within technology to share the network interface

Mgmt
VM VM VM VM
VM

Hypervisor

CPU and Chipset

Baseboard

VF VF VF VF PF VF VF VF VF PF

Switching & PCIe SR-IOV Switching & PCIe SR-IOV

NIC with HSR/PRP and PCIe SR-IOV, #1 NIC with HSR/PRP and PCIe SR-IOV, #2

PTP Redundant Network PTP Redundant Network

Clock To Single Network Clock To Single Network
Support Conversion Support Conversion

Ethernet Ethernet Ethernet Ethernet

Port A Port B Port A Port B

External Redundant Network Pairs

Figure 6: Server with NIC Cards that support HSR/PRP networking, PTP clock sync, and SR-IOV.
Multiple such NIC cards can connect to PCIe card slots of server to support redundant networking requirements and virtualized
network interfaces for efficient access to networks from multiple VMs. The combination of HSR/PRP, PTP, and SR-IOV
technologies are expected to achieve a high level of both performance and scalability for VPR.
8
efficiently and reliably to the VMs within the 2.5.1 VPR Redundancy and High Availability
VPR server. The NIC would thus implement Each protection relay shall meet the dependency
the combination of HSR/PRP and SR-IOV definition basically the ability to operate or trip
capabilities as shown conceptually in Figure 6. for a fault within its protective zone. In many
This approach routes network packets directly cases the protection engineer will specify a
to the required VM using PCIe SR-IOV virtual scheme for a substation or a powerline that has
functions to meet the reliability and latency multiple relays which operate in a way that they
requirements for VPR. have overlapping responsibility such that if any
one relay fails to trip on a fault, another relay in
2.5 Hypervisor Selection the system will detect that fault and trip. In some
Hypervisors are the essential ingredient of the cases, the utility protection engineer may deploy
virtualized system as they are the software layer two complete protection relays in parallel with
that receives and conveys requests between the same algorithm on the same zone. Thus, a
the physical and virtual resources. Modern protection engineer uses her knowledge of the
hypervisors have modes and schedulers that grid to avoid the chance that a failure of a relay
provide various options for supporting the could result in a fault condition that damages
virtualization of real-time operating systems and equipment or creates a persistent safety hazard
real-time applications. to utility personnel or the public. A second case
where redundancy in the protection scheme is
When a hypervisor is installed directly on
required is for the rare but sometimes necessary
the hardware of a physical machine, it is
maintenance or replacement of protection
called a bare metal hypervisor. Bare metal
system components.
hypervisors may also be embedded into the
firmware enabling better hardware handling. The introduction of VPR offers new ways to
Other hypervisors approaches are possible economically add redundancy to the protection
including host-based hypervisors which actually scheme. Modern hypervisors allow deployment
run inside a host operating system of the of VMs in high availability cluster mode. The
physical machine. An example for a bare metal host server or VM management console will
hypervisor is VMWare ESXi. An example of a handle the redundancy and failover of the virtual
hypervisor installed with an operating system is machines with negligible or even zero downtime.
the TTTech Nerve platform which offers a Linux Hence a failure of a single VPR system will not
KVM hypervisor. affect the protection system operation.

Protection Protection Failover Protection Protection

Function 1 Function 1 Function 1 Function 1
Replication
Protection Protection Protection Protection
Function n Function n Function n Function n

VM 1 VM 2 Failover VM 1 VM 2

Replication
VM Hypervisor 1 VM Hypervisor 2

VPR Server 1 VPR Server 2

Figure 7: VM Hypervisor redundancy

Two VPR servers, each with VM Hypervisors, can be coordinated in a side-by-side redundant topology to add resiliency to a VPR-based protection system.
The OEMs can use the fundamental capabilities of virtualization in this way or with other topologies for enhanced performance, availability, and resilience.

9
2.5.2 Implementation of Digital twin performance and time requirements of the
More utilities are now fascinated with the idea of modern protection IEDs while adding the
implementing digital twins in the substation for redundancy, resiliency, digital twin, ease of
design reviews, testing, managing, and upgrading deployment, and other features mentioned
of the system. VPR infrastructure allows easy earlier in the paper.
implementation of digital twin for the protection
relays. Digital twin implementation allows the The Kalkitech VPR reference application has
simulation of the protection relays and real-time been designed, developed, and tested in
updates. With VPR, the simulated relay will have virtualized platform running in Intel Xeon based
the same configuration and receive the same data Server hardware. The reference application
from the field as the original relay. Thus, with VPR is available for the utility engineers and
the digital twin concept can be leveraged for fault system integrators to test and compare the
trouble shooting and replay based on IEC 61850 performance with the contemporary protection
model and Disturbance data from the field. systems. Figure 8 provides a map of a protection
sub-system including the powerline, current
and voltage sensors (CT and PT, respectively),
3. VPR Reference System the sampling and digitizing unit for the current
Kalkitech has created a set of a reference and voltage signals (merging unit, or MU), the
protection applications to pair with VPR Server breaker control unit (BCU), the process bus
hardware, hypervisors, and other components to network, the VPR node with internal details
create a complete VPR system to serve multiple exposed, the station bus. The set of software
roles in the market adoption: as a proof of components needed to configure and monitor
concept, a demonstration system, and to help the the protection system including the human-
industry to evaluate the capabilities, trade offs, machine interfaces (HMI), and a disturbance
and challenges posed by the VPR architectural recorder (DR) is shown to the right side of the
options. A key objective is to prove that multiple diagram. The VPR can be interfaced to the
protection relays can work in a virtualized utility’s supervisory control and data acquisition
environment meeting or even exceeding the (SCADA) system as well.

VPR (IED) Third-party/Vendor1 Signal

CT
Processor/MMS
IPC API SCADA/HMI
Protection Fx1
Station Bus (GOOSE / MMS/HTTPS)

Third-party/Vendor2
SV IEC 61850
MU IPC API Client
(xx sample/Cyc) Shared
Memory
Process Bus

Conditioning
BCU SV Subscriber Module 1 MMS Server Web HMI

Shared
Memory Configuration
GOOSE

Tool
IPC API
PT
DR Analyzer
DR Module
SV IPC API Settings File
GOOSE CID/SCD file

Electrical MU/BCU Process Bus VM Station Bus Software

Boundary Boundary
Field Hypervisor Front End

Figure 8: VPR Reference Application within the chain of components for VPR. Some details of the SW modules in the VPR Server, and the connections between
those modules, are shown. One MU/BCU pair is shown in the diagram, but it is expected that once the system is mature, a single VPR server could serve more
than 100 MU/BCU pairs.

10
Within the VPR node, the details shown in Substations. Reference protection functions
Figure 8 include the SV subscriber to collect available for evaluation includes Distance
the streams of digitized current and voltage Protection (21), Transformer Differential
signals from the process bus, conditioning Protection (87T), Time Delayed Phase
module(s) for digital signal processing to extract Overcurrent Protection (51), Time Delayed
key parameters from the data streams, one or Derived Neutral Overcurrent Protection (51N),
more protection functions to detect abnormal Instantaneous Phase Overcurrent Protection
power system condition(s) as per their algorithm (50), Instantaneous Derived Neutral Overcurrent
and settings, and a disturbance recorder (DR) Protection (50N) and Breaker Failure
module for capturing detailed event data. Protection (50BF).

In Figure 8, only one MU and BCU are shown, but

for a VPR server in the field it is expected that 3.1 Kalkitech VPR Framework
one VPR server (once mature) can handle data The Kalkitech Framework provides a set of
from 100 or more MU’s and BCU’s. Similarly, base ingredients which OEMs and utilities can
within the VPR node, a multiplicity of protection use to deploy an IEC 61850 VPR infrastructure
functions and conditioning modules would be to evaluate migration of their capabilities to
needed to support multiple MU’sand BCU’s. the virtualized environment using the APIs
provided. As such, the term “Third Party /
The VPR reference application is developed Vendor” in Figure 8 is used in the areas where
using IEC 61850 VPR infrastructure provided domain experts and developers from multiple
by Kalkitech. It provides basic infrastructure companies can insert their conditioning and
backbone for implementing (Virtual) Protection protection function components. Figure 9
Relays based on IEC 61850 for Digital shows a few models of how the Kalkitech VPR

Kalkitech Reference Protection Relay Protection Relay

Applications Vendor 1 Vendor 2

Protection Functions Protection Protection

Functions Functions
21 50 51 50BF
Conditioning
87T 50N 51N
Module

VPR API

Sampled Value Conditioning IEC 61850 Disturbance IEC 61850

Subscriber Module Server Recording Configuration

IEC 61850 VPR Infrastructure

Figure 9: Kalkitech IEC 61850 VPR Framework.

The VPR Infrastructure and Reference Applications for demonstration and verification of other components of a VPR system. The Kalkitech VPR
Infrastructure can also support OEMs’ algorithms and signal processing functions to create a complete VPR server. The protection functions are
labeled as per ANSI / IEEE Standard C37.2 Standard for Electrical Power System Device Function Numbers, Acronyms, and Contact Designations.

11
infrastructure and reference applications can be the MU over the process bus (i.e., 960 samples
used and co-exist with third party components. per second for each current and each voltage
signal inside the VPR as compared to 4800
The framework provides mandatory sample per second for each data stream from
communication modules for VPR the MU, in a 60Hz AC grid system). These 16
implementation. Protection developer or vendor sample per cycle data are used for phasor
can integrate the modules using the API library estimations on each stream: recursive Discrete
provided by the framework. The framework Fourier Transform (DFT) algorithms are used
already has the IEC 61850 implementations like to calculate the fundamental and up to the 5th
IEC 61850 9-2 LE Sampled Value Subscriber, IEC harmonic. Sequence components are used for
61850 /MMS Server, GOOSE Publisher module, the estimation of signal frequency.
Disturbance recorder and API library to use the
functions. It also has a signal processing module Figure 10 shows conceptually how the signal
that serves data to the reference protection conditioning recursively updates its estimations
functions. The data from this signal processing on each sample (e.g. 960 times per second
module can also be used by other protection in the case of a 60Hz grid frequency). This
modules or applications. approach provides both signal continuity and
rapid detection for large instantaneous changes
3.2 Signal Conditioning within the VPR to the signals.
Reference
3.3 Laboratory Setup for Testing of VPR
This section of the paper describes additional
details of the implementation of VPR that has The complete reference VPR application has
been tested. The SV receiver module extracts been tested and evaluated using a Protection
instantaneous values of voltages and currents Relay testing kit.
are shared with the signal conditioning module
The VM Hypervisor shall handle multiple
which interpolates and down samples the data
network cards to support process bus and
to adjust for frequency deviations between
station bus. Protection Relay test kit is used
the nominal frequency and the actual grid
for generating sampled values which serves
frequency. These down-sampled signals will
the purpose of Merging Unit. It also measures
have 16 samples per cycle for each data stream
the operating times based on receipt of
compared to the 80 samples per cycle from
respective GOOSE messages. VPR configuration
and monitoring is from another VM. This is
achieved over the station bus or by using VM
DFT-1 virtual network.
DFT-2
DFT-3

Figure 10: Sliding window concept used for phasor calculations.

The DFT algorithm executes at 960 times per second in a 60 Hz grid system.

12
3.4 Evaluation of VPR Reference Application according to industry practices to cause the
Kalkitech has evaluated the reference protection function under test to operate. The
applications for functionality and end-to- Doble test kit and the VPR server are networked
end latency in their own lab. In these tests, a using a RuggedCom switch. The VPR reference
Dell PowerEdge R740 server with Intel Xeon applications receive and process the sampled
Gold 6248 R (24 Core) processor and 192 GB value data to generate a trip decision periodically
RAM hosted a VMWare ESXi Hypervisor and (approximately 960 times per second for most of
two virtual machines, as shown in Figure 11. the protection functions). When a fault condition
IEC 61850-9-2 LE compliant SV streams are is detected, the VPR reference applications
generated by Doble F6150sv test kit with a user- generate and publish a GOOSE message which
defined set of test cases representing the grid is received by the Doble test kit to determine the
current and voltage. The test cases are created operating threshold and time.

VM RUGGEDCOM Switch D-Link Switch

Management

Windows Laptop
SV/GOOSE
NIC-1

NIC-2

NIC-3

NIC-4

v Switch-0 v Switch-1 (a

eth0 eth1 eth0 eth1

VM-2 VM-1
Windows 11 OS Photon OS
(Test Computer) (VPR)

VMWare E SXi
VMWare ESXi
FO / MT-RJ

Doble
F6150sv

Intel Xeon Server Computer

to vSwitch-0

Shared memory API

Figure 11: Web HMI
Test setup showing the physical
devices and VM’s of the VPR Server. 21 50 51 50BF
a) The devices include the
Relay Test Kit
VPR Server, Doble relay tester Software
6150sv, network switches, and
management Laptop. 87T 50N 51N
8-1 MMS
Server
(b
b) Expanded views of the software IEC 61850 GOOSE
modules in the VM’s show the Explorer Pub/Sub
Shared memory API
connectivity and interfaces
between the modules and the Signal
VM-2 Conditioning
virtual switches of the ESXi
hypervisor.
IEC 61850 9-2 Disturbance
to vSwitch-1 SV Subscriber recorder
VM-1

13
The Breaker Failure function is a multi- Table 1 gives the protection function response
step function meant to take effect if an time when the VPR reference application is
initial overcurrent protection function tested in a Photon 4.0, 64-bit operating system
responds to an overcurrent condition, VM. The IEC-61850 sampled values have been
but the overcurrent condition is not generated using the Doble kit at a sampling
eliminated (which can be a sign that the rate of 4800 samples/sec for a 60Hz frequency
circuit breaker IED or the circuit breaker power system. The test pattern for generating
itself are not operating as expected). the trip signal are step functions starting at
The Breaker Fault function has re- zero RMS current level and rising to a level
trip time and then a backup trip time of 10 A RMS which is 200% of the 5A RMS
parameters which govern the speed at trip threshold.
which the backup protection message
will be generated and published to the 3.4.1 Response Time Dependence on HW
process bus. Allocation to VM
The ESXi hypervisor provides configuration
The Transformer Differential Protection settings to tailor the portions of the physical
function requires two SV data streams to hardware that is allocated to each VM on the
be compared to test whether essentially system. For a robust design, one criterion is
the amount of energy entering the to ensure that the behavior of the protection
transformer and the amount of functions did not depend significantly on the
energy exiting the transformer are resource allocation to the VM over a wide range
approximately equal. Because the of configurations. As an initial test of this, a
comparison is required, the algorithm Kalkitech reference application VM (Photon OS)
requires additional time to respond. was provisioned with between 256MB and 8GB
of RAM and from 1 core to 5 cores of the CPU.
In this range of configurations, the 50, 50N, 51,
and 51N protection functions were confirmed
to have no significant variation in response time.
Below 256 MB of memory the VM functioning
The response time is measured by the test kit as can degrade. A set of this data is presented in
the delta time from the start of the fault until the Table 2 where a Photon OS VM with 4 CPU cores
receipt of the associated GOOSE message. The and varying memory allocation is presented.
response time quoted below include the network Additional in-depth testing of these and
transit time(s), which is measured separately to other ESXi hypervisor configuration settings
be in the order of 1 millisecond round trip. combinations is ongoing.

Protection functions response time (milliseconds)

No. of SV
50 51 50N 51N 21 87T
streams

1 14 15 10 13 14 --

2 15 14 9 11 13 23

3 15 14 8 12 14 22

Table 1: Responses Time for Protection with 1 to 3 SV streams processed by one VPR system.
Protection Functions designed as per ANSI Standard codes: 50 - Instantaneous Phase Overcurrent Protection, 51 - Time Delayed Phase Overcurrent Protection,
50N - Instantaneous Derived Neutral Overcurrent, 51N - Time Delayed Derived Neutral Overcurrent, 21 - Distance Protection, 87T - Transformer Differential
Protection. In this figure, each SV stream is a simulated test pattern of current and voltage timeseries telemetry data streams of a three-phase power system
node (80 samples per cycle).

14
 Protection functions response time (milliseconds)

RAM 50 51 50N 51N

256 MB 11.6 12.0 11.7 12.1

512 MB 11.5 11.9 11.6 12.0

1 GB 11.5 11.6 11.6 11.7

2 GB 9.7 11.0 9.7 11.1

4 GB 10.5 11.2 10.7 11.3

8 GB 11.1 11.9 11.2 12.0

Table 2: Response time of protective functions with different memory level allocations. The test pattern is for RMS current is a step function from 0 to 200%
of threshold.

3.4.2 Response Time versus Fault Magnitude The protection function design with iterative
The following tests were conducted to understand moving window phasor estimation at 16
the dependency of the protection algorithm on samples per cycle are expected to respond
the fault magnitude. These tests were carried out faster when the magnitude of the fault current is
for the instantaneous overcurrent protection (50) increased. The data shown in Table 3 and Figure
in a virtual machine running Photon Linux with 12 confirms this expectation.
1CPU core and 4GB RAM.

Shot No. 0-10A 4-10a 4-15A 4-20A 4-25A 4-50A

1 17.29 8.13 5.73 4.79 4.69 5.21

2 13.44 8.44 10.1 5 6.77 6.77

3 9.69 10.31 8.23 7.6 5.42 5.21

4 14.38 10.94 6.77 6.77 5.42 3.65

5 10.42 6.15 7.6 5.63 4.69 4.06

6 11.98 8.13 6.15 4.79 7.29 5.1

7 13.75 6.25 5.73 8.96 6.88 4.69

8 10.1 6.77 5.63 5.42 4.69 5.21

9 9.58 10.1 6.77 7.19 7.19 3.13

10 10.42 7.92 6.15 4.69 5.52 5.73

Mean 12.105 8.314 6.886 6.084 5.856 4.876

STD-DEV 2.55 1.69 1.41 1.47 1.07 1.05

STD-DVE/Mean 21% 20% 21% 24% 18% 22%

Table 3: Response times for varying fault magnitude (trip threshold is 5A). Each scenario was tested 10 times with the values, mean, and standard
deviation reported.

15
 Instantaneous OvercurrentTrip Delay - Mean Value RMS current
step function is from 80% threshold of X*100% of threshold
Doble Tester (SV) to Doble Tester (GOOSE) 9

8
Trip delay in milliseconds from

0
0 2 4 6 8 10 12

Multiple of Threshold for Overcurrent

Figure 12: Mean response time with respect to amplitude change in current.
The trip threshold is 5A in all conditions; each condition was tested 10 times and the standard deviation of the measurements is approximately 0.2 of the mean
value. The data shows the expected trend of reduced delay to pick up the fault as the fault current level increases.

The absolute values of delay approaching ¼ has brought in the much-needed modularity
of a 60Hz power cycle, using widely adopted that makes possible the consideration to
relay test equipment, and with a realistic VPR build software-defined systems that can
reference software stack, and using relatively take advantage of the maturing virtualization
conservative CPU and memory resources from technologies applied to real-time and
the server platform under test, is intended to industrial applications.
be a first step for advancing the VPR concept
towards definition, trial use, standardization, At the same time, the industry is facing the
and eventually broad commercialization by rapid increase in renewable energy resources
the industry. on the grid (and especially distributed
renewable energy resources); which bring
variability and altered power flow patterns
4. The Path Ahead not seen before at-scale. These renewable
Any protection system must meet the basic resources are projected to become not only
requirements of Selectivity, Speed, Simplicity, the single largest category of bulk energy
Reliability, Economy. These qualities are produced, but at unprecedented levels in
best achieved by using proven algorithms absolute terms. The virtualized and software-
and methods on top of the best technology defined industrial systems paradigm presents
available at a given period in time. The advent an opportunity for the power utility sector
of IEC 61850 has standardized the modeling to re-imagine their protection and control
of protection functions, the way data is shared systems’ implementations and to use these
both at process level and station level and technologies to maintain the ubiquitous power
how data can be shared between protection quality and availability that sustains our society
modules over a communication channel. This and economy.

16
Hardware vendors such as Dell, Advantech, Work continues for advanced time
Crystal Rugged, Moxa, and others have synchronization and new techniques for
built commercial hardware devices that network virtualization. Industry collaboration
are candidates for the VPR solution. These is essential for standards to configure and
devices adhere to IEC61850 and IEEE 1613 manage protection functions in a virtualized
specifications and are compatible with advanced environment. Ultimately industry collaboration
networking interface cards for process bus and and standards for performance and
station bus implementations. interoperability certification will be created in
sharing and processing data across multiple
The adoption of VPR has the promise to modules, processes, and applications.
consolidate the large number of different
purpose, make/model, size, and age of discrete 4.1 Learn More about Intel Technologies
devices (hundreds of them in many substations)
to a small stack of redundant servers which are 1. Hardware and Software for Real-Time
standardized and interoperable. Minimization Computing https://www.intel.com/content/
and standardization of hardware can lead www/us/en/developer/articles/technical/
to more efficient and effective management real-time-computing.html
and maintenance not to mention ease of 2. Introduction to Cache Allocation
replacement and upgrade as well as dramatic Technology in the Intel® Xeon® Processor
space savings. The hardware devices can E5 v4 Family https://www.intel.com/
fit within the management responsibility content/www/us/en/developer/articles/
of the enterprise IT teams of the utility, for technical/introduction-to-cache-allocation-
example, leading to better total lifecycle technology.html?wapkw=cache%20
cost. Cybersecurity and planned software allocation
upgrades are among the services for which the
established IT practices can be applied.

Notices & Disclaimers

Performance varies by use, configuration and other factors.
No product or component can be absolutely secure.
Your costs and results may vary.
Intel technologies may require enabled hardware, software or service activation.
Intel does not control or audit third-party data. You should consult other sources to evaluate accuracy.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or
its subsidiaries. Other names and brands may be claimed as the property of others. 0522/TB/VC/PDF

17