Cyber Infrastructure for the Power Grid

Dr. Anurag K. Srivastava, Dr. Carl Hauser, Dr. Dave Bakken

Data Management and Computing (Computation)

February 26, 2014, and beyond

Administrivia
Contact info

Email: [email protected],, Office: EME 55, Office hours: right after I

lecture, or by email appointment.

Any comments or questions so far I can answer?

Meet someone across the EE-CS chasm!

A little background on me

Text: Chapters 1-2 of [CDKB5] Distributed Systems: Concepts and Design,

5ed, Addison Wesley, May 2011, ISBN 0-13-214301-1

Schedule for This Segment

2/26: #1--Utility IT infrastructures; control center
structure & software;
2/24-3/3 Take Home midterm exam
3/3: #2--CIMs, IEC 61850 and 61970
3/5: #3--Distributed computing #1: basics (CDKB5 chap
1)
3/10 #4: Distributed computing #2 (CDKB5 chap
2.1+2.3)
3/12 #5: Overview of Fault-Tolerant Computing
(CDKB5 2.4)
Then Assignment #3 given out
(Prof. Hahn starts on security on 3/24 and 3/26)
3/31: #6--WAMS data delivery requirements and
mechanisms, including NASPInet and GridStat

Caveats on the big picture

On our generalized architectural descriptions:
All generalizations are false, including this one.
Pascal
Result: NERC functional model

Power industry moves very, very slowly esp. on

information and communication technologies (ICT)

Use of advanced ICT in the power grid is way behind that in

other industries
E.g., middleware best practices in other industries for 15-20
years, barely used in the electricity sector

Utilities

are trying hard to be first to be second

Jeff Dagle (WSU MSEE), Chief Electrical Engineer, PNNL

Utilities are quite willing to use the latest technology,

so long as every other utility has used it for 30 years
unknown

Reading for Computation Segment

1.

Felix F. Wu, Khosrow Moslehi, and Anjan Bose. Power System Control Centers: Past,
Present, and Future. Proceedings of the IEEE, 93(11), Nov. 2005, 1890-1908. Sec
1-4, 6, 8. Note: inside the wsu.edu domain downloading is free, outside $$.

2.

EPRI, Common Information Model Primer, 1ed, 2021. Sec 4+5.

3.

IEC 61850, http://en.wikipedia.org/wiki/IEC_61850

4.

CDKB5: George Coulouris, Jean Dollimore, Tim Kindberg and Gordon Blair,
Distributed Systems: Concepts and Design Fifth Edition, published by Addison
Wesley, May 2011 (Chapters 1-2). WSU bookstore; www.cdk5.net

5.

D. Bakken, R. Schantz, and R. Tucker. Smart Grid Communications: QoS Stovepipes

or QoS Interoperability?, Grid-Interop 2009 (best connectivity paper), Denver,
November 18, 2009.

6.

M. Armbrust et al. Above the Clouds: A Berkeley View of Cloud Computing,

Technical Report No. UCB/EECS-2009-28, February 2010.

7.

D. Bakken, A. Bose, C. Hauser, D. Whitehead, and G. Zweigle. Smart Generation and

Transmission with Coherent, Real-Time Data. Proceedings of the IEEE (Special Issue
on Smart Grids), 99(6), June 2011, 928-951. Preprint (if IEEE paper not accessible).

Notes: #1-2,5-7 are in Blackboard: Assignments/Reading; #6-7 optional but may be

useful and/or interesting.

Todays Content
Quick review of earlier big picture & Intro (I)
Control Center Evolution (II)

Conventional Control Centers (III)

Changing Environment (IV)
Modern Distributed Control Centers (VI)

Future Control Centers (VIII)

Notes
Much overviewed in first segment (Prof. Srivistava),
going fairly fast for context
Skipping cause in rest of this segment and others:
Enabling [ICT] Technologies (V)
Emerging [ICT] Technologies (VII)

Power System Structure

Hydro

Gas or CC

Nuclear

Coal

Basics
Generation &
transmission
Substations &
transformers
Control centers

http://tcip.mste.illinois.edu/
Credit: Jim McCally, Iowa State

control centers

Introduction (I)
Impetus to have control centers came from big 1965
blackout in northeastern US
Energy Management System (EMS) resulted
Fairly

crude ICT by 1980s standards

Core largely obsolete by 2000 ICT standards

Likely future: cloud-based infrastructure (grid

services in paper)
Very

fast and reliable data acquisition

Many more apps, with intelligence pushed in or towards
substations
Much more sharing of data between apps and between
utilities and others (?DHS)

Control Center Evolution (II)

1950s: analog communications deliver analogs to
analog computers
Load

frequency control (LFC) and economic dispatch (ED)

Unit commitment (UC): schedules start/top for generators

1960s: SCADA and RTUs slowly begin to be used

1965 blackout recommendation: use digital
computers more extensively and effectively for RT
1970s: power system security/reliability: ability
of system to withstand disturbances/contingencies.
Custom

computers general purpose computers

running EMS
SCADA installed in substations and distribution feeders
distribution management systems (DMS)
SCED: security-constrained economic dispatch

Control Center Evolution (cont.)

Host of supporting functions in grid to
ensure reliability
Real

power reserves
Ancillary services: additional capabilities
(often inter-utility) to help ensure reliability
(e.g., load following)

Control Center Evolution (cont.)

Deregulation (of generation): mid-90s
Old:

vertically-integrated utility with all 3 fundamental

roles in its territory
New
Competition for generation
LSE: load serving entity
ISO/RTO to run transmission markets for LSEs
CC expanded from reliability-based EMS to also include
economic/business functionality
Two-level structure of CCs in market

Two types of markets (complex, balance in RT!):

Bilateral

contracts between suppliers and consumers

Auction market: generators submit bids
Also markets for day-ahead, real-time balancing (5min),
and ancillary services

Control Centers (CC) in the Market Environment

BMS:

business
management
systems
ISO/RTO operates
market
MOS: market
operations system
Market

participants:
Gencos, Trancos,
LSEs, ISO/RTO, etc

EMS and BMS Interactions

SCUC:

securityconstrained unit
commit ~ 5min
SCED: securityconstrained
economic
dispatch

Conventional Control Centers (III)

Looking from two angles: functions and architecture

Functionality provided by power system application

programs (apps)
Two main kinds of functions:
Extensions

of the first EMSs: reliability

Data acquisition
Generation control: NERC role balancing authority (BA)
Network (security) analysis and control: NERC role
reliability authority
State estimation
Contingency analysis / security analysis
BMSs

for market operations: economics

(skipping details . BMSs are not the focus of this overview)

Control Center Functions

Note: ERP and Data Warehouse discussed later

Control Center Architecture

SCADA designed for vertically-integrated utilities and
very little useful data in substations
Star

topology feeding into one computer: inflexible

Today: substation automation adds many IEDs that

could be quite useful
Lots of data starting to be used by many different
applications (historically in proprietary formats)
Different

subsets used by different apps

Multiple copies have to be coordinated, synchronized,
merged into DBs
CIMs to the rescue (next lecture)

Conventional Control Center Architecture

Substations

have
LANs connecting
IEDs to RTU
Formerly: wires!
61850 .

RTU

P2P network
link to front end
(FE) to CC LAN
SCADA developed
when utilities
vertically
integrated
monopolies;
structure same

Changing Environment (IV)

Very dynamic situation
Vertically-oriented

utility to deregulation
Divestitures, mergers, acquisitions
Control centers with non-contiguous territories
New participants coming, some disappear
all generalizations are false
But now all control centers have to cope with changing
business architecture (ouch!) above, below, and with
peers

Coarsely-integrated EMS, SCADA, DA, morphs

towards a coherent enterprise architecture
Need flexibility in many dimensions and at many
levels of the EAs components. modularity .
scalability, expandability.

Changing Environment (cont.)

Control centers sensor and derived data contain
(literally) a wealth of information
Key

issue: how to cleanly integrate (again, CIMs help

here)

Part of soln: sophisticated SW from other domains

ERP: enterprise resource planning
AKA

enterprise resource management (ERM)

Manages all aspects of business: production planning,
material purchasing, maintaining inventories, supplier
interactions, payroll, customer service, etc.

Data warehouse: carefully integrated (and scalable)

collection of utilitys data
Bundled

with statistical tools, e.g. SAS

Can repackage easily for other subsets, applications, etc

Integration Needs of Control Centers

CCs

have to
integrate in new
ways
Vertically (both
above and below)
Horizontally (new!)

E.g.,

NASPI (North
American
Synchrophasor
Initiative) and
similar emerging

Modern Distributed Control Centers (VI)

Distributed Computing System: system where apps
and services are spread out over multiple computers
that communicate only via message passing
I.e.,

no shared memory
A centralized EMS and BMS suite in one room matches
(oops)

Better definition/term: decentralized

E.g.,

distributed control logic pushed out of substation

E.g., RAS/SPS/SIPS not in CC or otherwise centralized

Current trends in distributed control systems

Separation

of SCADA (data acquisition), EMS, and BMS

IP-based distributed SCADA (DNP3)
Standard (CIM)-based distributed data processing
Middleware-based distributed EMS and BMS apps

Distributed Data Acquisition

Data

acquisition not
tightly coupled with EMS
Now: IP/DNP3 SCADA
Soon: NASPInet/GridStat

Future Control Centers (VIII)

Data acquisition: much higher amount & range of
Rate

of collection

DFRs: 720+ Hz
F/Net: 1440 Hz
Some home sensor apps ~5K Hz; broader uses?
Latency

required
Criticality of application
Quantity of data Geographic distance shared over
Degree of sharing (#subscribers) per sensor

Far from one size fits all data acquisition and

delivery!

Future Control Centers (cont.)

Security monitoring and control (SE, contingency
analysis, etc) based on steady-state models
Reason:

inadequate data acquisition (mainly) and

computational power in CCs
Result: only preventative control
Possible: analytical tool for emergency control by system
operators (or DHS/NERC?) .

Future Control Centers (cont.)

This (and much more) possible by
Cleanly

decomposed and packaged application logic

Separate management of data acquisition, managing
computational resources, information security, etc.
Cloud computing (grid services in paper)
ARPA-E project GridCloud with Cornell and WSU
Let

the power folks and ICT folks both do what they are
good at!

This clean separation enables

new technology (both apps and ICT)
to develop much fasterj
It worked for the internet!

email WWW phone...

Though it now needs QoS at the waist

ethernet PPP

SMTP HTTP RTP...

TCP UDP
IP

CSMA async sonet...

copper fiber radio...

The Brave New World

Cloud computing (IF made mission critical!) enables

Greatly

expanded applications (and app providers)

Federations of enterprise grids, ancillary services,
Dynamic sharing of computational resources of all IEDS
Distributed services for data acquisition and data
processing (aggregation, error collection, etc)
Distributed control center apps expressed in terms of
layers and services (~SOA)
Use of standard cloud computing architectures
and tools to manage ICT resources

I dont think were in Kansas

anymore, Toto!
Opportunities abound for a lot of fun and impact for newlyminted EE and CS grads working in this field!