GEC Engineering Training 2010

for
Advanced SCADA engineering'

FAST/TOOLS
fundamentals and
performance tuning

Venue: YEF-NLS
Date: Mar. 1, 2010 (Mon) - Mar. 5, 2010
(Fri)
Conducted by: YHQ GEC ETC Support
Copyright Yokogawa Electric Corporation

GEC Engineering Training 2010

Advanced SCADA engineering

Overview

Basics FAST/TOOLS
Major performance topics:

Hard disk
DUR queues
Locking
Item common

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Basics FAST/TOOLS

FAST/TOOLS tries to keep an up do date

internal copy of the field situation
This copy is kept in memory in items
Mimics etc. display these items
Data transfer is done mostly event
based

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Basics FAST/TOOLS
Display, history,
alarming, objects,
etc
Item area

Item

Item

Item
Item

Item

EQP

PLCs & RTUs

Copyright Yokogawa Electric
Corporation

GEC Engineering Training

Basics FAST/TOOLS, data streams

Data between FAST/TOOLS processes

can be transferred by:
Items (Tag based data blocks in memory)
DUR messages (Post office mechanism
between processes)
Files on disk (Mostly configuration data and
history data)

Processes can subscribe for item

changes and/or file changes
Each item/record change will be send to the
process by a DUR message (= event)

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Basics FAST/TOOLS, data streams

Mimic
= DUR message

Setpoint:
69.3

Item area
Item

Item

EQP

PLCs & RTUs

Copyright Yokogawa Electric
Corporation

GEC Engineering Training

Overview

Basics FAST/TOOLS
Major performance topics:

Hard disk
DUR queues
Locking
Item common

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk

Introduction to hard disk:

Disk performance
Fragmentation
Caching
Conclusion

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, performance

Data transfer speed:

Typical 70Mbyte / Sec

Access time:
Typical 7200 Rpm (One cycle each 9 ms)

Read speed:
When reading one big contiguous file:
70Mbyte / Sec
When reading random blocks of 1Kbyte
average: 0.2Mbyte / Sec

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, fragmentation

Disk defragmentation: partition can be filled

at random

Defrag disk will make files contiguous again

Example ...

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, caching

Cache files in disk cache: retrieving data

from cache is much faster
Cache is controlled by windows.
Normally last read file(-parts) will be in
it
Only RAM not claimed by processes will
be used for cache

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, conclusion

Reading is slow when small bits and

pieces are read
Reading is faster when complete
contiguous large files are read
Caching can speedup file access
enormous

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, FAST/TOOLS databases

Each database has one data file (.d01) and

one or two key files (.k01 and .k02). The files
are bucket based (1-8Kb).
A data file bucket contains an amount of slots.
Each slot can contain one record. Records are
not sorted.
A key file bucket contains an amount of keys
which are sorted.
Reading one record means one or more key
bucket reads and one data bucket read.

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, FAST/TOOLS databases, file

structure

Key file

Data file

Fd

Af

Af

Fc

Bz

Fc

Hx

Fa

Ht

Ve

Fd
Ga
Ht

Example ...
Copyright Yokogawa Electric
Corporation

GEC Engineering Training

Bz

Hard disk, optimalization

Defragmentation
Cache
Bucket size
Record order

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, optimalization,

defragmentation

Is standard windows functionality

Has risk when done on live system
Defragment at least always once after
completing the configuration

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, optimalization, cache

FAST/TOOLS delivers haccache program to

load files in cache (version 9.01 and higher)
Can run once and/or cyclic
Default used before starting redundant
systems
Can also be used for caching executables
etc.
Exercise ...

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, optimalization, bucket size

Using bigger buckets reduces number of key

index levels
Program isforder (since 9.02SP1) can change
bucket size of a file to max size (=8Kb)
Newly created PROCESS/FAST databases
(since 9.02SP1) will always have 8Kb buckets
Exercise: What is the actual bucket size
for ...\tls\dat\opc_attrib.dat?

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Hard disk, optimalization, record order

Sorting records in the order of the most

used key speeds up reading
Program isforder (since 9.02SP1) can
sort a file on a key number
File itmsav.dat can be sorted by giving
itmdbg command save all items to
disk

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Overview

Basics FAST/TOOLS
Major performance topics:

Hard disk
DUR queues
Locking
Item common

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues

DUR message queues:

Each FAST/TOOLS process has a message queue
Messages in a queue are normally handled serial
by the process
All queues are allocated in a common data area
(DUR common)
Each queue is allowed to grow up to a maximum
buffer size (in Kbytes)
When total claimed queues reaches total
common area size you are in trouble
DUR common area size can be changed in
DUR.SUP (>32MB needs move item common)
Exercise: Change queue size to 11MB

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, common area

Process queue
headers

Mess.

Common area

Mess.
Mess.

Mess.

UMH
Mess.
ITM

Mess.

Mess.

EQP1

Mess.

AAA

Mess.

Other data
headers
Data

Data

ISM data

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Data

DUR queues, common area high used

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, process list

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, overflows

When maximum queuing for a process

is reached future messages will be lost
(failed messages).
Receiver process can get information
about failed messages.
Use of failed message information and
reaction method is different per process.

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, overflow reactions

Typical reactions on failed messages:

Ignore or only count missing events. (When
alarm printer buffer overflows at 1000
alarms the operator is only happy)
Detect missing events and repair by total
scan action. (When some intermediate
values did not reached the mimic there is no
real problem)

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, overflows

Critical lost events:

Real data is lost e.g.:
Current alarm overview not correct
Value change not stored in item history

This should be avoided.

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, non critical processes

UMH Converts error codes to messages

UMHLOG Distributes error messages
LGHUMH Stores error messages in history
UMHLOGTOO* Shows error messages in a
window
ALDPRT Prints alarm lines on a printer
For all these processes loss of some data during
a big burst is not so important

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, inherent repair processes

WFLIDI/CWIITM
Handles item changes to mimics
On overflow a total scan is performed

WFLADI/CWIALD
Handles alarm changes to mimics
On overflow a refresh of the page is performed

ALDACO
Generate sound when a new alarm is detected
In many configurations there is only one type
of sound, so by overflow the sound is already
generated by the previous alarms
In case of different sound per priority there
can be a problem
Copyright Yokogawa Electric
Corporation

GEC Engineering Training

DUR queues, inherent repair processes

ITMCPM
Handles host-frontend item exchange
Will recover after overflow

DURM_*
Used for messages from other nodes
Will never come above 80%

Node queues (more or less)

Most processes communicating over node
boundaries can deal with lost messages

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, critical processes

ADTSTO
Stores audit trail events
Will only overflow when bad configured

ALM
Generates alarms out of status changes
Queue overflow means missing alarms and
not correct current alarm overview
In some system startup scenarios normal
but should than be corrected with ALMUPD
Queue should be increased when max
usage above 80%
Copyright Yokogawa Electric
Corporation

GEC Engineering Training

DUR queues, critical processes

ALH
Stores current and chronologic alarm
information on disk
Will deliver alarm information when browsing
in alarm windows
Queue overflow will result in wrong alarm
history
Process can be delayed by browsing with an
ASA selection
Queue should be increased when max usage
above 80%
Possible to split process in chronologic and
current part
Copyright Yokogawa Electric
Corporation

GEC Engineering Training

DUR queues, critical processes

ITHEEX
Stores item event/event type history on disk
On queue overflow some history samples
are not stored
Each event results in at least 2 disk writes
Disk throughput can be to small
Use for items which are changing often the
event/item storage type
Increasing queue will only help in bursts
situations

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, critical processes

ITM
Handles:
Item configuration requests
Remote set/get item requests
Retain last value disk updates

Queue overflow most times caused by

wrong retain last value configuration
Increase queue if max used above 80%

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, critical processes

OPCMAN
Handles:
Configuration classes and objects
Write changes on attributes to disk

Can get problems when:

Attributes used as items (often changed)
Bad performing DURM communication lines

Changing queue size mostly does not help

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, critical processes

OPCEXE
Runs the PROCESS/FAST objects
Can get bursts when e.g. equipment
communication goes down or up
Performance on bursts improved a lot in
9.02SP1
When max queue above 80% increase
queue size or split in more OPCEXE
processes

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, critical processes

EQP*
Handles communication with PLCs and RTUs
Queue used for output item commands
For each incoming change on input/output
items also a message is queued which
results in a burst during communication
recover
Increase queue when max used above 80%

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

DUR queues, configure queue sizes

Configuration of queue size can be done:

For many processes:
In the setup file of the process or tool
E.g.:
for OPCEXE3 in OPC.SUP
For EQP123 in EQP123.SUP

For node queues:

In the setup file DUR.SUP, parameter
NCB_MAX_QUEUE

Example ...

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Overview

Basics FAST/TOOLS
Major performance topics:

Hard disk
DUR queues
Locking
Item common

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Locks

Common data areas have locking

mechanism
Collisions can occur when multiple
processes wants to lock same common
area
If collisions occur more 250/second and
system has performance problems
contact ETC

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Locks, display

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Overview

Basics FAST/TOOLS
Major performance topics:

Hard disk
DUR queues
Locking
Item common

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Item common

Item common area:

Contains part of the item information like
actual value, status, quality, etc. (Rest stored
on disk)
Can be examined with ITMDBG, choice 2
Should have at least 10% space
Use is rather static after configuration
completed
Size can be changed in ITM.SUP
Exercise: How many space left has the item
common area?
Copyright Yokogawa Electric
Corporation

GEC Engineering Training

Item common, display usage

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Database limits

DATABASE/FAST should be checked on

the following limits
Maximum users
Maximum files
Maximum files size

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Database limits, maximum users

With debugger ISMDBG the actual number of

users and maximal number of users can be found
Give command ISMDBG and give choice 1
Nearly at the end line xxx entries used out of yyy is
shown.
Xxx is actual number of entries
Yyy is maximal number of users

Maximal number can be changed in

database_start, switch u during start of process
lockmgr
Maximum value was about 2000 but increased in
9.02SP1 to about 16000
Copyright Yokogawa Electric
Corporation

GEC Engineering Training

Database limits, maximum files

With debugger ISMDBG the actual number of open

files and maximal number of open files can be found
Give command ISMDBG and give choice 2
Nearly at the end line xxx entries used out of yyy is
shown.
Xxx is actual number of entries
Yyy is maximal number of users

Maximal number can be changed in database_start,

switch f during start of process lockmgr
Maximum value was about 2000 but increased in 9.02SP1
to about 16000

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Database limits, maximum file size

The maximum size of a single database

file is 2GB
This is caused by a limitation of the seek()
and tell() interface of the standard C library
For WINDOWS there is an other interface
available which we are going to use in one of
the next releases
For UNIX we will try to find an other interface

Copyright Yokogawa Electric

Corporation

GEC Engineering Training

Thank you for your

attention

Commitment means building the future to

last
Copyright Yokogawa Electric
Corporation

GEC Engineering Training