IEC / IEEE

PMU (Pilot Project) PMU (IPP) Central PDC Regional PDC Local PDC

A wide area monitoring system with more than 5000 PMUs deployed in India
 by Hubert Kirrmann, Solutil, Switzerland and William Dickerson, Arbiter Systems, Inc, Canada

39
39

IEC/IEEE
Precision Time
Protocol Profile

Precision Time Protocol

Electrical utilities depend increasingly on precise clock synchronization to
detect grid instability and protect the infrastructure. The IEC and IEEE collaborated
on a profile of the Precision Time Protocol that suits the needs of the power utility
industry and that is integrated in the IEC 61850 utility communication suite. The
resulting standard IEC/IEEE 61850-9-3 provides sub-microsecond time accuracy and
supports seamless redundancy.
Time Synchronization in the ent places at exactly the same time. To this effect, the phase of Hubert Kirrmann was born in Stras-
Grid This allows applying simple Kirch- current is measured at strategic bourg, France. He graduated in electro-
Electrical utilities have been one hoff’s law to sense abnormal situa- positions in the grid by Phasor technical engineering from ETH Zurich.
of the most demanding applica- tions. To this purpose, a sampling Measurement Units (PMUs). India He taught at the District University of
tions regarding time precision. His- accuracy of some microseconds is is deploying a wide area monitor- Bogota, Colombia, at the Swiss Federal
torically, the electrical grid was the needed. Since only relative time ing system with more than 5000 Institute of Technology both in Zurich
primary provider of time for the matters here, the time distribution PMUs. Synchrophasor transmis- (ETHZ) and Lausanne (EPFL). He retired
synchronization of public clocks for differential protection could be sion over PSN (packet switch- in 2013 from the EPFL as honorary pro-
(synchronous clocks) and this re- done either via a dedicated link, e.g. ing networks) is specified in IEC fessor. He retired after 33 years at the
mains true even today as the grid carrying a 1 PPS signal. This works 61850-90-5. ABB Research Center in Baden, Switzer-
operators ensure that the number well within a substation, but for The phase measurement in each land. He works since as consultant for
of periods at the end of the day differential line protection between PMU is time-stamped with respect his own company, Solutil. He is president
matches the 86400 seconds times substations, it is impracticable to to absolute time within a few mi- of the Swiss IEC TC65 (industrial auto-
the grid frequency (this is a fre- span dedicated synchronization croseconds. Indeed, a phase error mation), member of IEC SC65C WG10 /
quency distribution, so the clocks lines. Rather, the communica- precision of 0,1° corresponds to TC57 WG15 and editor of several IEC and
still need to be set). tion network was used under the time error of 5 µs on the sampling. IEEE standards.
To track back disturbances, assumption that the delays were A clock accuracy of 1 µs is deemed William Dickerson is Chairman of the
for instance to reconstruct the predictable and symmetrical. This more than sufficient for this appli- Board of Arbiter Systems, Inc. in Paso
sequence of events that led to a was the case with the old analogue cation, since the instrument trans- Robles, California. Bill received his BSEE
blackout, a precise time-stamping telephone lines, and also SDH/ formers and the filtering introduce from Washington University in St. Louis
of events with respect to absolute SONET networks, but this virtue a larger error. and his Masters in Business Administra-
time is needed. In this application, went lost with the digital data com- Early PMUS were synchro- tion from the University of Michigan.
the time distribution over the net- munication packet switching net- nized by radio receivers listening Bill worked at Hewlett-Packard in Palo
work using the SNTP [13] protocol works, which need exact time to to broadcast of atomic clocks, e.g. Alto, CA and Spokane, Washington from
with an accuracy of some 10 ms resynchronize the samples. WWV in the USA or DCF-77 in 1979 until 1986, when he and his current
was sufficient. Synchronization to absolute Europe. The devices in the substa- partners bought Arbiter Systems, Inc.,
With the introduction of digi- time became indispensable to de- tion were connected in star fash- which is the industry leader in precise
tal substations, e.g. based on IEC tect grid instabilities. Grid collapses ion to the receiver by dedicated measurements for power systems and
61850-9-2, a precise time distribu- announce themselves by variations lines, e.g. according to the IRIG-B in accurate, reliable timing products for
tion with a better accuracy is need- in the grid frequency, so comparing protocol. the power industry. Bill was the original
ed. Differential protection schemes the phases of voltage and current at Synchronizing by a GPS receiver product designer for Arbiter’s GPS timing
protect assets by detecting unbal- strategic nodes of the grid allows to in each device has become cost- product line and the Model 1133A Power
ances in the current flow. Current detect an incipient black-out and effective, but remains technically Sentinel, the industry’s most accurate
and voltage are sampled at differ- take measures. impracticable. The reason is that Phasor Measurement Unit (PMU).

PAC.SEPTEMBER.2016
 by Hubert Kirrmann, Solutil, Switzerland and William Dickerson, Arbiter Systems, Inc, Canada

40

2
IEC/IEEE
utilities do not trust the GPS sig- Figure 1 shows an example of PTP time transmission
nal. Some GPS clocks, especially in time synchronization in a substation.
GPS
locations with restricted sky view, The primary reference comes from
have shown degraded performance the wide area network (e.g. an SDH/
depending on the satellites that are SONET or MPLS network), where a
visible. The antenna must also be number of atomic clocks maintains
MC
deployed outside of the substation the absolute time. In case of WAN 0
2μs t0
if the substation is entrenched or disruption, a local rubidium atomic
subterranean. Especially in north- clock maintains the time until WAN
50μs TC
ern countries, the GPS signal can recovers. A GPS receiver is used for
Precision Time Protocol

be interrupted by aurorae. The sig- consistency check and to adjust leap

5μs
nal could be spoofed, presenting a seconds in UTC time. This diversity is t0 52
cyber-attack risk. also a protection against cyber-attacks
So, the solution that imposed that spoof time. As help for commis-
200μs TC
itself, is to use atomic clocks, e.g. sioning, the master clocks can gener-
rubidium clocks, in dedicated net- ate 1 PPS signals to check that all IEDs
work nodes. The time is transmit- are correctly synchronized. 4μs t0 257

ted over the wide area network Figure 1 shows three different
(WAN) that controls the grid, as types of bay that need synchroniza- OC t = to + 261
explained in IEC TR 61850-90- tion: classical bays with no process
12. Similarly, the time signal is bus, bays with merging units to classi- one hears Big Ben striking noon, it is noon past
distributed within the substation cal instrument transformers and bays some seconds, depending where one is.
using the same Ethernet as for the with a process bus and distributed In radio transmission, the distance between
data communication, as explained digital instruments. the sender and the slave clocks and the signal
in IEC T R 61850-90-4. This Principle of PTP paths must be known to estimate the propaga-
The IEC and IEEE makes that an IED can be used as a Precise time distribution relies on tion delay. In long-wave radio such as DCF-77,
PMU. In addition, the LAN trans- the estimation and correction of all surface propagation is assumed while in short-
mission eliminates the dedicated transmission delays between a master wave radio such as WWB the (quite variable)
collaborated on 1 PPS or IRIG-B links within the clock giving the absolute time and a ionospheric path is considered. GPS greatly
substation. slave clock synchronized to it. When improves the precision by precise location of
a profile of the the slave clock with respect to the satellites

Precision Time
1 Time distribution in a substation network
and estimation of the signal path delay with
an atmospheric model. In data networks, the
atomic clock reference clock
radio reference clock delays consist of propagation delay over the
(back-up master) (GNSS)
back-up and calibration medium (link delay, about 5 µs/km) and for-
Protocol that warding delay of the packets in the network
GPS
SCADA nodes, bridges or routers (residence delay, in
Rb back-up the order of milliseconds depending on traffic).
SCADA
suits the needs The IEEE 1588 Precision Time Protocol
OC OC (PTP) distributes both TAI (atomic time) and
of the power MC MC MC UTC (legal time). It was standardized in 2008
1 PPS and adopted by IEC as IEC 61588 in 2009.
station bus 1588 (for short) lets a master clock broadcast a
utility industry Sync message that carries the exact time when
TC TC TC the Sync left the egress port of the master. The
and that is Sync message travels through the network.
Each network node that receives a message
protect control protection
over a port forwards it on all its other ports,
protection
integrated in the & control BC & control BC after correcting the time to account for its
OC OC residence delay and the link delay (per port), as
process bus process bus Figure 2 shows. A device that relays the Sync
IEC 61850 utility
and corrects it is called a transparent clock
(TC). TCs can be used in LANs (over Ethernet)
MU MU
communication and in WANs.
The transparent clocks are normally in-
tegrated in bridges (switches). However, the
suite. conventional bay merging unit bay digital instruments
Note: Abbreviations are defined in Figure 8

PAC.SEPTEMBER.2016
 41
IEEE 802.1D bridge functionality
does not apply to 1588. So even
dence time that the peer indicated
in its Pdelay_Resp message, i.e.
3peer
Link delay calculation by
delay measurement
ports blocked by the Rapid Span-
ning Tree Protocol send (modified) (t4 - t1) - (t3 - t2) transparent clock
link
Sync messages. 2 link

The measurement of the resi- This method assumes a sym- Pdelay_Req Pdelay_Req
dence time in each node requires metrical delay. If an asymmetry is t12
link t11
t12
Pdelay_Resp
that a port can precisely record the known beforehand, it can be com- t13
Pdelay_Resp
t13
t14 t14
time at which a Sync enters the port pensated for. Here also, a two-step
t11 t14
and the moment the (modified) correction can be applied, and in
Sync message leaves the port. To this case the peer returns its resi-
this effect, a time-stamping mecha- dence delay in a subsequent Pde-
Pdelay_Req Pdelay_Resp
nism is implemented directly in the lay_Resp_Follow_Up message.
Ethernet physical interface (PHY) 1588 also supports a different
as Figure 4 shows. kind of clock nodes, called bound-
To this effect, the TC captures ary clocks (BCs). The boundary Figure 3:
t12 t13
the egress time when the first bit clocks were the historical solution
of the header after the preamble is in 1588v1 before the advantages of To correct for
sent, calculates the difference be- transparent clocks became obvious.
tween egress and ingress time, adds A boundary clock is a network the link delay,
the link delay to it, sums it to the node in which one port is synchro- reference and of the frequency stability of the
former correction field, modifies nized by another master clock and local oscillator. each port of
the correction field and finally ad- synchronizes the local clock. The Each transparent clock introduces an error in
justs the checksum of the frame all other ports of the boundary clock the timestamping, since its local quartz frequen-
while the frame is being transmit- act as master in their respective re- cy is not infinite and the local oscillator has only a clock node
ted (Figure 5). This is called a one- gion. So a boundary clock separates a limited stability. For implementation reasons,
step correction. regions within with a time distri- there are several quartzes and therefore quanti- calculates
Since correcting a frame on the bution domain, each region having fication errors appear. And finally, the quality of
fly can only be done in hardware, its own master. The master at the the local oscillator of a TC is lower than that of the link delay
1588 foresees also a two-step cor- top of the hierarchy is called the the grandmaster, even if it is properly synchro-
rection in which a subsequent Fol- grandmaster. nized (tuned to the frequency of the master). to its peer
low_Up message transmits the While BCs do not offer the same So inaccuracy increases with the number of
correction, which is not considered accuracy as TCs, because a chain of TCs and BCs clocks in series. Figure 7 shows the
here to simplify. clocks need to be synchronized, time error histogram of the different clocks in by a request-
To correct for the link delay, they are effective to separate the function of their position in the Sync path. For
each port of a clock node calcu- network into islands should the the purpose of power utility automation, the response
lates the link delay to its peer by a main synchronization fail. They PMUs and other demanding devices must be lo-
request-response message, called however cannot serve as protocol cated so that the accuracy at their location is still message,
a Pdelay_Req and Pdelay_Resp, as translator, e.g. between profiles of sufficient.
Figure 3 shows. 1588. Figure 6 shows the elements The IEC-IEEE 61850-9-3 Standard called a
A port measures the link de- of a 1588 time distribution scheme 1588 allows a number of options, including
lay to its peer by calculating the Clock Accuracy 1-step/2-step correction, end-to-end/peer-to-
time difference between sending Each clock introduces a certain peer delay measurement, layer 2/layer 3 opera- Pdelay_Req and
the Pdelay_Req and receiving the inaccuracy. The master clock accu- tion, different LANs, etc. but does not specify
Pdelay_resp, and deducing the resi- racy depends on the quality of the performance. Pdelay_Resp.

4 Time-stamping in a network node 5 Sync frame in 1588 with one-step

link delay residence delay link delay correction
subtract ingress time-stamp
Transparent Clock from egress time-stamp, add link delay and
add to former correction
PHY PHY PHY PHY
link link correct checksum
bridging logic timestamp
(e.g. cable) (e.g. cable)
point

Xtal FCS body correction header preamble

2240 ns 1760 ns
reference plane reference plane @100 Mbit/s

PAC.SEPTEMBER.2016
 by Hubert Kirrmann, Solutil, Switzerland and William Dickerson, Arbiter Systems, Inc, Canada

42
table 1 IEC/IEEE 61850-9-3
IEC/IEEE
The IEC/ IEEE 61850-9-3 worst-case approach was chosen.
Standard is a profile of 1588 for There is no ongoing check of main parameter
the power utility automation, i.e. clocks degradation, except for the
it is a subset of 1588 except for the grandmaster accuracy, which can Parameter Value
(optional) redundancy method. It degrade due to loss of reference Profile identifier 00-1B-19-00-02-00
is derived from the profile in 1588 signal. A grandmaster is supposed Clocks MC, BC, TC, OC
Annex F (Ethernet) and default to maintain its accuracy during 5 Media Ethernet Layer 2
profile Annex J.4 (peer-to-peer). s after loss of reference signal (e.g. Topology full-duplex IEEE 802.3
The main parameter of IEC/ GPS) Transmission Multicast messages
IEEE 61850-9-3 are summarized Time accuracy depends on the Multicast 01-1B-19-00-00-00
Precision Time Protocol

address (all but pDelay)

in Table 1. interval between Syncs. Other 01-80-C2-00-00-0E (peer delay)
The 1588 does not specify per- 1588 profiles use shorter intervals, VLAN Allowed, not recommended
formance. IEC/IEEE 61850-9-3 but it was shown that 1 s intervals Ethertype 0x88F7 (1588)
specifies by how much the differ- are optimum. The grandmaster Subtype 0 (version2)
ent components may degrade the also issues an Announce message Path delay measurement peer-to-peer only
time inaccuracy, so as to guarantee that indicates its characteristics; Correction 1-step and 2-step, may be mixed at will
sub-microsecond accuracy after 15 here also, 1 s was chosen to account All settable, default = 0,
Clock domains
TCs or three BC. for a master transfer within 5 s. domain 93 recommended
The network engineer knows Redundancy Principle Announce interval 1s
from this information where to Substation automation sys- Sync interval 1s

place the grandmaster and the tems require a very high availabil- Pdelay_req interval 1s
Figure 6 shows Announce timeout 3s
slaves that require high accuracy. ity. Therefore, LAN redundancy
The manufacturer guaranteed Default and IEC 62439-3 when redun-
is required. IEC 61850 specifies to MBCA dancy is used.
the elements clock inaccuracy is also part of the use the redundancy scheme of IEC Parallel operation defined in IEC 62439-
Backup master
object model of each clock, which 62439-3 [8] Parallel Redundancy 3, multiple masters allowed
is also readable when the devices Protocol) (PRP) and High-availabil- Priority 1 128
of a 1588 time
are in place. But discovering dur- ity seamless redundancy (HSR). Priority 2 128
ing commissioning that a substa- PRP operates on the principle Medium converters 50 ns
inaccuracy
distribution tion network presents intolerable that the nodes are attached to two GM inaccuracy 250 ns
inaccuracy accumulation can cause separate, completely independent TC inaccuracy 50 ns
scheme. high costs in re-engineering, so a LANs operated in parallel. Such BC inaccuracy 250 ns
Syntonization Implicitly required

6 Elements of a 1588 time distribution network

Holdover time
Signalling
5s
Not specified
1) IEC62439-3 Annex E
PT elements Management 2) IEC 61850-90-4
3) other
grandmaster (role): Top Region
top level clock of the time do- MIB Defined in IEC 62439-3 Annex E
Designated Master (DC)
main master of the top region, master-enabled OC (back-up clock of domain) Local Time Optional ATOI (I588 16.3)
defines grandmaster identity (currently slave) first to become master (after
Rb BMCA) if the current master Security IEC 62351
master port GMC OC stops sending Announce
sends Sync & Announce:
sets "sourcePortIdentity"
DMC a doubly attached node sends a frame over both
ingress port slave or passive port
for this region LANs simultaneously; the frames travel through
Transparent Clock (TC)
both LANs; a receiving node takes the first frame
Ordinary Clock (OC) can
forwards and corrects PTP TC OC take the role of master or slave and discards the duplicate.
messages
eggress ports Figure 8 shows the principle of PRP, when
Boundary Clock (BC) has a slave port a master clock sends a Sync to a slave clock. PRP
slave port in the upper region Sync & Announce carry
and a master (or passive) port in the MAC address of the defeats the principle of 1588 because the delay
the lower region
BC BC
slave port master,
(not that of the
that a Sync frame experiences is different when
master port grandmaster) travelling over one LAN or over the other LAN.
Region B send Sync and Announce and the
OC "sourcePortIdentity" The Sync frames will have different correction
of the master
fields and therefore will not be duplicates of each
slave port TC TC Region C other. Accepting both frames for synchronization
Hybrid Clocks (HC) would cause a significant jitter in the control loop
combine a TC and a OC, of the oscillator of the slave clock. When 2-step
HC have two or more ports
If this link is established, the two regions merge and OC correction is used, it would be difficult to pair the
one boundary clock's master port becomes either slave HC HC
or passive, according to the BMCA
bridging nodes
Sync and the corresponding Follow_Up.

PAC.SEPTEMBER.2016
 43
Therefore, IEC 62439-3 An- station bus operates with PRP and The precision time protocol
nex A modifies the 1588 principle the process bus operates with HSR.
exclusively for PTP messages. A In this case, there are four Sync allows additional services, such as
slave treats the master clock seen
on LAN A and LAN B as two differ-
frames circulating every second.
This does not cause a significant deterministic data transmission by
ent clocks and applies a Best Master
Clock algorithm to select its master.
traffic load.
Since a slave selects its master,
time slotting, if this should prove
At the same time, the slave su- several masters could be active at necessary.
pervises the other LAN to detect the same time in each LAN. While
failures of the other path. Since the this brings little in terms of avail-
identity of the master is the same ability, this diversity allows a con-
over both LANs, the slave uses ad- sistency check and protects against
ditional time quality information
to select the master, for instance
cyber-attacks trying to spoof time.
Conclusion
7 Degradation of time accuracy
PTP path
the one with the smallest correc- With IEC/IEEE 61850-9-3 and
tion field or the smallest jitter. (see IEC 62439-3, the power industry
Figure 8). has a standardized, high perfor-
IEC 62439-3 also applies this mance, high-availability network TC TC SC
principle to HSR as Figure 9 shows. capable of distributing sub-micro-
grandmaster transparent transparent slave
Within the HSR ring, the sync second time synchronization for clock clock clock clock
frames travel in both directions and the most demanding applications.
are corrected by each HSR node. The precision time protocol al- local clock error
(pdf)
Since corrected Syncs frames are lows additional services, such as
no duplicates any more, they are deterministic data transmission by
total error
removed from the ring by their time slotting, if this should prove (pdf)
source, using the back-up mecha- necessary.
nism of HSR. Each node selects Return on experience is sought,
the Syncs from one direction and indeed, the UCA organized a suc-
uses the other direction to check
healthiness.
cessful plug-fest and interoper-
ability test in Brussels in November
9PRPPrinciple of parallel redundancy with
(top) and HSR (bottom)
An HSR ring can be connected 2015 that showed that devices of
to a PRP double LAN, as Figure 9 different manufacturers could in-
shows. This could be the case if the teroperate smoothly. GMC
Sync A Sync B

8 Principle of parallel redundancy with PTP

Sync A MC Sync B LAN A LAN B

interlink A interlink B
link relay link relay residence
relay

RedBox RedBox
A BC HC BC B

LAN_A LAN_B
link relay Sync A

Sync BA Sync BB Sync BA Sync BB

residence link relay
HSR ring
relay link relay
B A B A B A B A
Sync A Sync B

HC HC HC HC
OC

PAC.SEPTEMBER.2016