Abcs Of Zos System Programming Paul Rogers

Alvaro Salla download

https://ebookbell.com/product/abcs-of-zos-system-programming-
paul-rogers-alvaro-salla-48713658

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Abcs Of Zos System Programming Paul Rogers Alvaro Salla

https://ebookbell.com/product/abcs-of-zos-system-programming-paul-
rogers-alvaro-salla-48713636

Abcs Of Zos System Programming Paul Rogers Alvaro Salla

https://ebookbell.com/product/abcs-of-zos-system-programming-paul-
rogers-alvaro-salla-48713638

Abcs Of Zos System Programming Paul Rogers Alvaro Salla

https://ebookbell.com/product/abcs-of-zos-system-programming-paul-
rogers-alvaro-salla-48713640

Abcs Of Zos System Programming Paul Rogers Alvaro Salla

https://ebookbell.com/product/abcs-of-zos-system-programming-paul-
rogers-alvaro-salla-48713642
Abcs Of Zos System Programming Paul Rogers Alvaro Salla

https://ebookbell.com/product/abcs-of-zos-system-programming-paul-
rogers-alvaro-salla-48713648

Abcs Of Zos System Programming Paul Rogers Alvaro Salla

https://ebookbell.com/product/abcs-of-zos-system-programming-paul-
rogers-alvaro-salla-48713660

Abcs Of Zos System Programming Paul Rogers Alvaro Salla

https://ebookbell.com/product/abcs-of-zos-system-programming-paul-
rogers-alvaro-salla-48713662

Abcs Of Zos System Programming Paul Rogers Alvaro Salla

https://ebookbell.com/product/abcs-of-zos-system-programming-paul-
rogers-alvaro-salla-48713664

Abcs Of Zos System Programming Karan Singh Paul Rogers

https://ebookbell.com/product/abcs-of-zos-system-programming-karan-
singh-paul-rogers-48713666
 Front cover

ABCs of z/OS System

Programming: Volume 5

Base and Parallel Sysplex, GRS, RRS,

ARM, sysplex failure management

System Logger, z/OS system

operation

GDPS, zSeries availability

Paul Rogers

ibm.com/redbooks
International Technical Support Organization

ABCs of z/OS System Programming: Volume 5

May 2011

SG24-6985-02
 Note: Before using this information and the product it supports, read the information in “Notices” on
page ix.

Third Edition (May 2011)

This edition applies to version 1 release 12 modification 0 of IBM z/OS (product number 5694-A01) and to all
subsequent releases and modifications until otherwise indicated in new editions.

© Copyright International Business Machines Corporation 2005 - 2011. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
The team who wrote this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Chapter 1. Base and Parallel Sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Evolution to a Parallel Sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 SYStems comPLEX or sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 The sysplex symmetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Sysplex philosophy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5 Single system image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 Parallel Sysplex workload balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.7 Sysplex software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.8 Sysplex hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.9 Sysplex Timer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.10 Server Timer Protocol (STP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.11 Coupling facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.12 Message time ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.13 Coupling facility LPARs and CFCC code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.14 Coupling facility links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.15 Sysplex overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
1.16 Cross-system coupling facility (XCF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.17 Base sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.18 XCF application, member, and group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
1.19 XCF services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
1.20 XCF signaling paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
1.21 XCF channel-to-channel connection (CTC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
1.22 XCF using coupling facility list structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
1.23 XCF exploiters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
1.24 Sympathy sickness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
1.25 Sysplex couple data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
1.26 Other couple data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
1.27 Parallel Sysplex with couple data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
1.28 Format utility for couple data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
1.29 Base sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
1.30 Parallel Sysplex. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
1.31 Cross-system extended services (XES) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
1.32 Sharing environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
1.33 Coupling facility structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
1.34 DB2 cross invalidation (1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
1.35 DB2 cross invalidation (2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
1.36 Level of Parallel Sysplex recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
1.37 User-managed rebuild. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
1.38 User-managed duplexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
1.39 User-managed duplexing rebuild . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

© Copyright IBM Corp. 2005 - 2011. All rights reserved. iii

 1.40 System-managed rebuild . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
1.41 System-managed CF duplexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
1.42 CF structure rebuild. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
1.43 Parallel Sysplex availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
1.44 CF configuration examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
1.45 Parallel Sysplex exploiters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
1.46 Defining the sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
1.47 IEASYSxx PARMLIB definitions for sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
1.48 IEASYSxx PLEXCFG parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
1.49 IEASYSxx GRS parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
1.50 CLOCKxx parmlib member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
1.51 COUPLExx PARMLIB member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
1.52 Consoles in a sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
1.53 Multisystem consoles in a sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
1.54 Sysplex operation and management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
1.55 Displaying CF information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
1.56 Display XCF information (1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
1.57 Display XCF information (2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
1.58 Display XCF information (3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
1.59 Display XCF signaling paths (1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
1.60 Display XCF signaling paths (2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
1.61 SETXCF command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
1.62 Managing the external timer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
1.63 Removing a system from the sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
1.64 Sysplex failure management (SFM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
1.65 Parallel Sysplex complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
1.66 Requirements of SFM policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
1.67 SFM implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
1.68 SFM policy parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

Chapter 2. System Logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

2.1 System Logger terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
2.2 Log stream recording media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
2.3 z/OS System Logger benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
2.4 Settings for System Logger services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
2.5 Creating and formatting LOGR couple data set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
2.6 LOGR policy definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
2.7 Managing log data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
2.8 Log stream in a coupling facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
2.9 System Logger XRC support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
2.10 DASD-only log streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
2.11 System Logger services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
2.12 System Logger exploiters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
2.13 SMF recording to logstreams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
2.14 SMF recording to SYS1.MANx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
2.15 SMF on System Logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
2.16 Where System Logger stores data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
2.17 Customizing SMF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
2.18 Which SMF records to record . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
2.19 System Logger log streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
2.20 Dumping SMF data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
2.21 Dumping SMF records - log streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
2.22 Dumping selective SMF records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

iv ABCs of z/OS System Programming: Volume 5

Chapter 3. Resource Recovery Services (RRS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
3.1 Introduction to Resource Recovery Services (RRS) . . . . . . . . . . . . . . . . . . . . . . . . . . 160
3.2 Two-phase commit protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
3.3 Two-phase commit process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
3.4 Two-phase backout process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
3.5 Resource Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
3.6 Set up RRS in the z/OS system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
3.7 RRS exploiters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
3.8 RRS commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
3.9 Command output from RRS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
3.10 Display Logger information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
3.11 XCF information about RRS structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
3.12 ATRQSRV batch utility support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
3.13 RRS archive logging and ATRQSRV utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

Chapter 4. Global resource serialization (GRS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

4.1 GRS introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
4.2 Resource access and GRS queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
4.3 Resource scope: Step, system, systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
4.4 GRS macro services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
4.5 RESERVE macro service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
4.6 Resource name list (RNL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
4.7 GRSRNL=EXCLUDE migration to full RNLs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
4.8 GRS configuration modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
4.9 GRS ring configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
4.10 GRS ring topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
4.11 GRS star configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
4.12 GRS star topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
4.13 GRS star highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
4.14 GRS star configuration planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
4.15 GRS star implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
4.16 Define GRS lock structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
4.17 Parmlib changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
4.18 GRS ring to GRS star . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
4.19 GRS star complex overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
4.20 Global ENQ processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
4.21 Global DEQ processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
4.22 Contention notification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
4.23 GQSCAN request for global resource data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
4.24 ISGGREX0 RNL conversion exit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
4.25 ISGGREX0 conversion exit flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
4.26 Shared DASD between sysplexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
4.27 ISGNQXITFAST and ISGNQXIT exits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
4.28 GRS star operating (1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
4.29 GRS star operating (2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
4.30 Global resource serialization latch manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

Chapter 5. z/OS system operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

5.1 Planning z/OS operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
5.2 Operating a z/OS environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
5.3 z/OS console types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
5.4 IBM 2074 console support controller. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
5.5 2074 console support controller configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
5.6 OSA integrated console controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246

Contents v
 5.7 Multisystem consoles in a sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
5.8 Sysplex operating environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
5.9 Support for multisystem management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
5.10 Message processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
5.11 Message flow in a sysplex environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
5.12 Command flow in a sysplex environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
5.13 Console-related parmlib members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
5.14 Display console status information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
5.15 Display system requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
5.16 Display all defined EMCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
5.17 Display information about an EMCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
5.18 Defining and changing console characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
5.19 The hardcopy medium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
5.20 z/OS operations log (OPERLOG) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

Chapter 6. Automatic restart management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

6.1 Automatic restart management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
6.2 ARM environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
6.3 Create an ARM couple data set and policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
6.4 ARM restarts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
6.5 Modifying batch and STC jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
6.6 Registering with ARM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
6.7 ARM element states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
6.8 ARM restart methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
6.9 Restart on the same system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
6.10 Restart on different systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
6.11 Group restart overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
6.12 ARM exit facilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
6.13 ARM SMF records. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

Chapter 7. Geographically Dispersed Parallel Sysplex . . . . . . . . . . . . . . . . . . . . . . . . 293

7.1 Business continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
7.2 Disaster recovery objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
7.3 SHARE disaster/recovery tiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
7.4 Tier 0 - No off-site data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
7.5 Tier 1- Pickup truck access method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
7.6 Tier 2 - PTAM and hot site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
7.7 Tier 3 - Electronic vaulting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
7.8 Tier 4 - Active secondary site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
7.9 Tier 5 - Two-site two-phase commit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
7.10 Tier 6 - Zero data loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
7.11 Database restart versus recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
7.12 GDPS introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
7.13 GDPS overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
7.14 Need for data consistency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
7.15 An example of write dependency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
7.16 GDPS terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
7.17 GDPS/PPRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
7.18 Freeze policy options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
7.19 Planned reconfiguration support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
7.20 Unplanned reconfiguration support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
7.21 GDPS/PPRC prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
7.22 GDPS/XRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325

vi ABCs of z/OS System Programming: Volume 5

7.23 GDPS/XRC prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
7.24 GDPS HMC automation interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
7.25 Peer-to-Peer Remote Copy (PPRC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
7.26 PPRC data flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
7.27 PPRC volume states - synchronous mode of operation . . . . . . . . . . . . . . . . . . . . . . 335
7.28 GDPS/PPRC HyperSwap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
7.29 GDPS/PPRC HyperSwap goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
7.30 GDPS/PPRC HyperSwap prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
7.31 Extended Remote Copy (XRC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
7.32 XRC components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
7.33 XRC data flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
7.34 GDPS error recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
7.35 GDPS/PPRC multi-site example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

Chapter 8. Availability in the zSeries and z9 environment . . . . . . . . . . . . . . . . . . . . . 351

8.1 System availability terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
8.2 Continuous availability as a target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354
8.3 A structured approach to continuous availability design . . . . . . . . . . . . . . . . . . . . . . . 355
8.4 Decision criteria for a continuous availability design . . . . . . . . . . . . . . . . . . . . . . . . . . 357
8.5 Hardware - Environmental aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358
8.6 Hardware - Central processing complexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
8.7 Hardware - coupling facilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
8.8 Hardware - switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
8.9 Hardware - DASD and other devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372

Contents vii
viii ABCs of z/OS System Programming: Volume 5
Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not give you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION
PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring
any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs.

© Copyright IBM Corp. 2005 - 2011. All rights reserved. ix

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation in the United States, other countries, or both. These and other IBM trademarked terms are
marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US
registered or common law trademarks owned by IBM at the time this information was published. Such
trademarks may also be registered or common law trademarks in other countries. A current list of IBM
trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
CICSPlex® Language Environment® System Storage®
CICS® MQSeries® System z10®
DB2® MVS™ System z9®
DS6000™ Netfinity® System z®
DS8000® NetView® Tivoli®
Enterprise Storage Server® OS/390® TotalStorage®
ESCON® Parallel Sysplex® VTAM®
FICON® PR/SM™ WebSphere®
FlashCopy® Processor Resource/Systems z/Architecture®
GDPS® Manager™ z/OS®
Geographically Dispersed Parallel RACF® z/VM®
Sysplex™ Redbooks® z10™
HyperSwap® Redbooks (logo) ® z9®
IBM® RMF™ zSeries®
IMS™ S/390®
IMS/ESA® Sysplex Timer®

The following terms are trademarks of other companies:

ITIL is a registered trademark, and a registered community trademark of the Office of Government
Commerce, and is registered in the U.S. Patent and Trademark Office.

IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency
which is now part of the Office of Government Commerce.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

x ABCs of z/OS System Programming: Volume 5

Preface

The ABCs of z/OS® System Programming is an eleven-volume collection that provides an

introduction to the z/OS operating system and the hardware architecture. Whether you are a
beginner or an experienced system programmer, the ABCs collection provides the
information you need to start your research into z/OS and related subjects. If you would like to
become more familiar with z/OS in your current environment, or if you are evaluating
platforms to consolidate your e-business applications, the ABCs collection will serve as a
powerful learning tool.

The contents of the volumes are:

Volume 1: Introduction to z/OS and storage concepts, TSO/E, ISPF, JCL, SDSF, and z/OS
delivery and installation

Volume 2: z/OS implementation and daily maintenance, defining subsystems, JES2 and
JES3, LPA, LNKLST, authorized libraries, Language Environment®, and SMP/E

Volume 3: Introduction to DFSMS, data set basics, storage management hardware and
software, VSAM, System-Managed Storage, catalogs, and DFSMStvs

Volume 4: Communication Server, TCP/IP and VTAM®

Volume 5: Base and Parallel Sysplex®, System Logger, Resource Recovery Services (RRS),
global resource serialization (GRS), z/OS system operations, Automatic Restart Management
(ARM), Geographically Dispersed Parallel Sysplex™ (GPDS), availability in the zSeries®
environment

Volume 6: Introduction to security, RACF®, Digital certificates and PKI, Kerberos,

cryptography and z990 integrated cryptography, zSeries firewall technologies, LDAP,
Enterprise identity mapping (EIM), and firewall technologies

Volume 7: Printing in a z/OS environment, Infoprint Server and Infoprint Central

Volume 8: An introduction to z/OS problem diagnosis

Volume 9: z/OS UNIX® System Services

Volume 10: Introduction to z/Architecture®, zSeries processor design, zSeries connectivity,

LPAR concepts, HCD, and HMC

Volume 11: Capacity planning, performance management, RMF™, and SMF

Volume 12: WLM

Volume 13: JES3

The team who wrote this book

The current edition of this book was produced by a technical specialist working at the
International Technical Support Organization, Poughkeepsie Center.

© Copyright IBM Corp. 2005 - 2011. All rights reserved. xi

 Paul Rogers is a Consulting IT Specialist at the International Technical Support
Organization, Poughkeepsie Center. He writes extensively and teaches IBM® classes
worldwide on various aspects of z/OS. Before joining the ITSO 23 years ago, he worked in
the IBM Installation Support Center (ISC) in Greenford, England providing JES support for
IBM EMEA and the Washington Systems Center. He has worked for IBM for 43 1/2 years.

Previous editions of this book were produced by Paul and other technical specialists from
around the world working at the ITSO Poughkeepsie Center.

Alvaro Salla is an IBM retiree who worked for IBM for more than 30 years in large systems.
He has co-authored many Redbooks® and spent many years teaching S/360 to S/390®. He
has a degree in Chemical Engineering from the University of Sao Paulo, Brazil. Alvaro was an
author on both the first and second editions of this book.

The team who wrote the the first edition of this book included Paul Rogers, Alvaro Salla, and
the following specialists.

Paola Bari is an Advisory Programmer at the International Technical Support Organization,

Poughkeepsie Center. She has 23 years of experience as a systems programmer in
OS/390®, z/OS, and Parallel Sysplex, including several years of experience in WebSphere®
MQ and WebSphere Application Server.

Luiz Fadel is a Distinguished Engineer working for IBM Brazil. He has over 31 years of
experience with the S/390 and zSeries platforms. He is responsible for supporting most
zSeries customers in Latin America. He has contributed extensively to several Redbooks.
Luiz worked in the ITSO from 1977 through 1980 and from 1988 through 1991.

Andreas Horn is an IT Specialist in the ITS Technical Support Center in Mainz, Germany. He
holds a graduate degree in Electronic Engineering and joined IBM in 1999. He supports
clients with handling defect and non-defect problems in all base components of z/OS.

Redelf Janssen is an IT Architect in IBM Global Services (Technical Support Services Sales
and Consulting) in IBM Bremen, Germany. He holds a degree in Computer Science from the
University of Bremen and joined IBM Germany in 1988. His areas of expertise include IBM
zSeries, z/OS, and availability management. He has written Redbooks on OS/390 Releases
3, 4, and 10, and was one of the authors of ABCs of z/OS System Programming Volume 3.

Valeria Sokal is an MVS™ system programmer at an IBM customer. She has 16 years of
experience as a mainframe system programmer.

Thomas Stoeckel is a z/OS SW specialist at the EMEA Backoffice in Mainz, Germany. He

has six years of experience in BCP, Parallel Sysplex, RRS, and System Logger. His area of
expertise includes z/OS performance analysis. Before joining the z/OS team, Thomas worked
for 15 years at the large system SW support center for VM and VSE.

Now you can become a published author, too!

Here's an opportunity to spotlight your skills, grow your career, and become a published
author—all at the same time! Join an ITSO residency project and help write a book in your
area of expertise, while honing your experience using leading-edge technologies. Your efforts
will help to increase product acceptance and customer satisfaction, as you expand your
network of technical contacts and relationships. Residencies run from two to six weeks in
length, and you can participate either in person or as a remote resident working from your
home base.

xii ABCs of z/OS System Programming: Volume 5

 Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html

Comments welcome
Your comments are important to us!

We want our books to be as helpful as possible. Send us your comments about this book or
other IBM Redbooks publications in one of the following ways:
򐂰 Use the online Contact us review Redbooks form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
[email protected]
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYJF Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400

Stay connected to IBM Redbooks

򐂰 Find us on Facebook:
http://www.facebook.com/IBMRedbooks
򐂰 Follow us on Twitter:
http://twitter.com/ibmredbooks
򐂰 Look for us on LinkedIn:
http://www.linkedin.com/groups?home=&gid=2130806
򐂰 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks
weekly newsletter:
https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm
򐂰 Stay current on recent Redbooks publications with RSS Feeds:
http://www.redbooks.ibm.com/rss.html

Preface xiii
xiv ABCs of z/OS System Programming: Volume 5
 1

Chapter 1. Base and Parallel Sysplex

A sysplex has been available since 1990 when it was announced as a platform for an evolving
large system computing environment. Sysplex provides a commercial platform that supports
the nondisruptive addition of scalable processing capacity, in increments matching the growth
of workload requirements for customers, without requiring re-engineering of customer
applications or re-partitioning of databases. Sysplex also allows the computing environment
to reach almost unimaginable levels of continuous availability or 24 by 7. Sysplex technology
is built on existing data processing skills and runs existing applications with additional cost
savings.

A sysplex (SYStems comPLEX) is not a single product that you install in your data center. A
sysplex is a collection of z/OS systems that cooperate, using certain hardware, software, and
microcode, to process workloads, provide higher continuous availability, easier systems
management, and improved growth potential over a conventional computer system of
comparable processing power. This chapter is an overview of a sysplex, including the
following topics:
򐂰 Sysplex benefits, evolution, and philosophy
򐂰 Required software and hardware for sysplex
򐂰 Coupling facility
򐂰 The z/OS sysplex services components: XCF and XES
򐂰 Several types of couple data set (CDS)
򐂰 The sysplex configurations: base sysplex and Parallel Sysplex
򐂰 Sysplex exploiters
򐂰 Coupling facility structure rebuild and duplexing
򐂰 Coupling facility configuration and availability
򐂰 An overview of settings for sysplex
򐂰 Consoles and sysplex management

© Copyright IBM Corp. 2005 - 2011. All rights reserved. 1

1.1 Evolution to a Parallel Sysplex

Figure 1-1 Evolution to a Parallel Sysplex

Evolution to a Parallel Sysplex

A Parallel Sysplex is the most recent development in the evolution of IBM large systems.
Large system configurations have evolved from a single system to a Parallel Sysplex in the
following progression:
򐂰 Single system uniprocessor
A single copy of the z/OS (or its ancestor) operating system manages the processing of a
central processor complex (CPC) that has a single Central Processor (CP), also called a
CPU.
򐂰 Tightly coupled multiprocessors
A single copy (also called an image) z/OS operating system manages more than one CP
sharing the same central storage, thus allowing several transactions to have their
programs executed in parallel.
򐂰 Loosely coupled configuration
This configuration is when more than one CPC, possibly tightly coupled multiprocessors,
share DASD but not central storage. The CPCs are connected by channel-to-channel
communications and are managed by more than one z/OS images.
򐂰 Base sysplex
A base sysplex is similar to loosely coupled but the z/OS system and applications
programs use a standard communication mechanism for exchanging messages called
XCF. This configuration makes the management easier because it provides a greater

2 ABCs of z/OS System Programming: Volume 5

 degree of communication and cooperation among systems and a more unified system
image with a single z/OS console group to manage all components.
򐂰 Parallel Sysplex
A Parallel Sysplex has up to 32 z/OS systems, as in a base sysplex, but it contains a
coupling facility (CF). The coupling facility is a global and intelligent memory that provides
multisystem data sharing (with total integrity), workload balancing, high performance
communication, and many other advantages.

Table 1 on page 3 summarizes the characteristics in terms of availability, capacity, and

system management according to the system configuration.

Table 1 Evolution to a Parallel Sysplex

Capacity Continuous Systems
availability management

Single system Limited by the size of Single points of failure Easy.

uniprocessor the largest single CP. and disruptive changes.

Tightly coupled Limited by the Single points of failure Easy.

multiprocessors maximum number of and disruptive changes.
CPs in the CPC.

Loosely coupled Increased over tightly Increased over tightly Each system must be
configuration coupled. coupled. managed separately.
Complexity grows with
the number of systems.

Base sysplex Same as loosely Better than loosely Single z/OS console
coupled. coupled because of the group to manage all
cooperation. components.

Parallel Sysplex Ability to add Total (24 by 7), if there Multisystem data-
incremental capacity to are not multiple sharing capability,
match workload growth. concurrent failures. multisystem workload
balancing, enhanced
single-system image.

Chapter 1. Base and Parallel Sysplex 3

1.2 SYStems comPLEX or sysplex

Figure 1-2 Systems complex or sysplex

SYStems comPLEX or sysplex

Parallel and clustered systems initially found in numerically intensive markets (engineering
and scientific) have gained increasing acceptance in commercial segments as well. The
architectural elements of these systems span a broad spectrum that includes massively
parallel processors that focus on high performance for numerically intensive workloads, and
cluster operating systems that deliver high system availability.

Parallel Sysplex clustering

Parallel Sysplex clustering contains innovative multisystem data-sharing technology, allowing
direct concurrent read/write access to shared data from all processing images in a parallel
configuration, without sacrificing performance or data integrity. Each image is able to
concurrently cache shared data in a global electronic memory through hardware-assisted
cluster-wide serialization and coherency controls. This in turn enables work requests
associated with a single workload, such as business transactions or database queries, to be
dynamically distributed for parallel execution on nodes in a sysplex cluster, based on available
processor capacity. Through this state-of-the-art cluster technology, the power of multiple
z/Series processors can be harnessed to work in concert on common workloads, taking the
commercial strengths of the z/OS platform to improved levels of competitive price
performance, scalable growth, and continuous availability. Prior to the Parallel Sysplex, S/390
(now called System z®) customers had been forced to contain capacity requirements of a
workload within technology limits imposed by the largest single symmetric multiprocessor
available (symmetric meaning all CPUs are the same).

4 ABCs of z/OS System Programming: Volume 5

1.3 The sysplex symmetry

Figure 1-3 The sysplex symmetry

The sysplex symmetry

You can think of a sysplex as a symphony orchestra. The orchestra consists of violins, flutes,
oboes, and so on. Think of each instrument as representing a different product (or
component) in the sysplex. The fact that you have several of each instrument corresponds to
having several images of the same product in the sysplex.

Think of symmetry in the orchestra in the following ways:

򐂰 All the violins (or whatever instrument) sound basically the same, and play the same
musical part.
򐂰 All the instruments in the orchestra share the same musical score. Each instrument plays
the appropriate part for that instrument.

Similarly in the sysplex, you can make all the systems, or a subset of them, look alike (clone
systems) and do the same work. All the systems can access the same database, and the
same library of programs, each one using the information it needs at any point in time. The
concept of symmetry allows new systems to be easily introduced, and permits automatic
workload distribution all the time, even in the event of failure or when an individual system is
scheduled for maintenance. Symmetry also significantly reduces the amount of work required
by the systems programmer in setting up the environment.

In an asymmetric sysplex, each system has its own software and hardware configurations, so
some of the system management benefits of being in a sysplex are lost.

Chapter 1. Base and Parallel Sysplex 5

1.4 Sysplex philosophy

Dynamic workload balancing

Data sharing

Incremental growth

Availability

Single system image

Different perspectives on single system image

Single point of control

Figure 1-4 Sysplex philosophy

Sysplex philosophy
A new violinist who joins the symphony orchestra receives a copy of the score, and begins
playing with the other violinists. The new violinist has received a share of the workload.
Similarly in a sysplex, if you add a system, the Customer Information Control System
(CICS®) OLTP transaction workload can be automatically rebalanced so that the new system
gets its share, provided you have set up the correct definitions in your sysplex.

Dynamic workload balancing

Theoretically, in the CICS OLTP environment, transactions coming into the sysplex for
processing can be routed to any system. CICS uses the z/OS component called Workload
Manager (WLM), along with CICSPlex® System Manager (CPSM), to dynamically route
CICS transactions. For this to happen in the sysplex, you need symmetry; the systems across
which you want to automatically balance the workload must have access to the same data,
and have the same applications that are necessary to run the workload. In other words, no
system affinity should be allowed.

Data sharing
We noted earlier that in the symphony orchestra, all the instruments share the same musical
score, each playing the appropriate part. You can think of the musical score as a kind of
database. Part of the definition of symmetry, as used in this book, is systems sharing the
same resources. An important resource for systems to share is data and programs, either in
the form of a database, or data sets. Symmetry through systems sharing the same database

6 ABCs of z/OS System Programming: Volume 5

and program libraries facilitates dynamic workload balancing and availability. The coupling
facility technology, together with the support in the database managers, provides the data
sharing capability.

You improve application availability by using products that provide data sharing with the
coupling facility technology, such as Information Management System Database Manager
(IMS™ DB), DB2® Universal Data Base for z/OS (DB2), VSAM RLS, or Transactional VSAM
Services (DFSMStvs).

Incremental growth
The conductor can add violins, or other instruments, to the orchestra one by one until the
desired effect is achieved. The conductor would not want to hire five more violinists if only two
are needed at the moment. A sysplex exhibits the same incremental growth ability. Rather
than adding capacity in large chunks, most of which might remain idle, you can add small
chunks closer to the size you need at the moment.

Also, the introduction of a new violinist is non disruptive. It is possible (although you might see
this only in the most novel of musical pieces) that the violinist could walk onto the stage in the
middle of the concert, take a seat, and begin playing with the others. There is no need to stop
the concert. Similarly, with a sysplex, because of symmetry and dynamic workload balancing,
you can add a system to your sysplex without having to bring down the entire sysplex, and
without having to manually rebalance your CICS OLTP workload to include the new system.

Continuous availability
If a violinist gets sick and cannot be present for a given performance, there are enough other
violinists so that the absence of one will probably not be noticeable. If a violinist decides to
quit the orchestra for good, that violinist can be replaced with another. A sysplex exhibits
similar availability characteristics. One of the primary goals of a sysplex is continuous
availability. You can think of availability from these perspectives: the availability of your
applications programs and the availability of your data.

With symmetry and dynamic workload balancing, your applications can remain continuously
available across changes, and your sysplex remains resilient across failures. Adding a
system, changing a system, or losing a system should have little or no impact on overall
availability. With symmetry and data sharing, using the coupling facility, you also have
enhanced database availability.

Automation plays a key role in availability. Typically, automation routines are responsible for
bringing up applications, and if something goes wrong, automation handles the application’s
restart. While automation does not play much of a role in our symphony orchestra, the need
for automation is quite important in the sysplex, for availability as well as other reasons.

A facility of z/OS called Automatic Restart Manager (ARM) provides a fast restart and
automatic capability for failed subsystems, components, and applications. ARM plays an
important part in the availability of key z/OS components and subsystems by decreasing the
mean-time-to-repair (MTTR), which in turn affects the availability of data.

For example, when a subsystem such as CICS, IMS DB, or DB2 fails, it might be holding
resources, such as locks, that prevent other applications from accessing the data they need.
ARM quickly restarts the failed subsystem; the subsystem can then resume processing and
release the resources, making data available once again to other applications. Note that
System Automation for z/OS (SA z/OS), an IBM product that provides automation of operator
functions such as start-up, shutdown, and restart of subsystems, has awareness of z/OS
Automatic Restart Manager, so that restart actions are properly coordinated. A sysplex is also
the framework that provides a single system image.

Chapter 1. Base and Parallel Sysplex 7

1.5 Single system image

GOAL

Figure 1-5 Single system image

Single system image

Think of all the violins in the symphony orchestra playing the same part. To the audience, they
might sound like one giant violin. The entire orchestra is cooperating to produce the music
that the audience hears. In this way, the audience perceives the orchestra as a single entity.
This is a good way to picture single system image in the sysplex. You have multiple images of
the same product, but they appear, and you interact with them, as one image. The entire
sysplex is cooperating to process the workload. In this way, you can think of the collection of
systems in the sysplex as a single entity.

Single system image is not a new concept. Many products already provide single system
image capability to some degree, or have plans to implement it in the context of commercial
enterprise-wide systems management. The important point is, single system image is a key
theme in a sysplex. Implementing symmetry in your sysplex facilitates single system image;
symmetry facilitates your ability to manage multiple systems in the sysplex as though they
were one system. Now, you have the best of two worlds, a logical centralized topology
implemented in a physically distributed one.

While single system image is the goal, different IBM and non-IBM products, and even different
components within products, are at different stages of development on this issue. Attaining
the goal depends on the installation choosing the right options on such products and
components.

8 ABCs of z/OS System Programming: Volume 5

Different perspectives on single system image
The single system image goal provides different advantages depending on your perspective.
The advantage for the end user is the ability to log onto an application in the sysplex, and to
be able to access that application without being concerned about which system the
application resides on.

For example, CICS uses the VTAM generic resources function, which allows an end user to
log on to one of a set of CICS terminal-owning regions (TORs), such as TOR1, TOR2, and
TOR3, through a generic name, such as TOR, thus providing single system image for VTAM
access to CICS TORs. With dynamic workload management, provided by CICSPlex SM (a
CICS component), the logons are then balanced across the CICS TORs; later, when the
transactions start to arrive, they will be dynamically distributed through transaction
application-owning regions (AORs) that are logically connected to the TOR at logon time. All
of this provides a single system image from an application perspective.

The advantage to an operator is the ability to control the sysplex as though it is a single entity.
For example, through commands with sysplex-wide scope, operators can control all the z/OS
images in the sysplex as though only one z/OS image existed.

When TSO/E is part of a sysplex and exists on multiple sysplex members, you can assign a
VTAM generic name to all TSO/E and VTAM application programs. A TSO/E and VTAM
application on one z/OS system can be known by the same generic resource as a TSO/E and
VTAM application on any other z/OS system. All application programs that share a particular
generic name can be concurrently active. This means that a user can log on to a TSO/E
generic name in the sysplex rather than to a particular system. The generic name can apply
to all systems in the sysplex, or to a subset.

Eventually, when all the necessary products provides single system image capability, the
result is greatly improved and simplified enterprise-wide systems management. Both IBM and
non-IBM products work towards this goal.

Single point of control

The conductor of the symphony controls the entire orchestra from the podium. The conductor
does not stand by the violins and conduct them for a while, and then run over and stand by
the flutes to conduct them. In a sysplex, an operator or a systems programmer should be able
to control a set of tasks for the sysplex from a given workstation.

The sysplex is a little different from the symphony orchestra in that single point of control in
the sysplex does not imply a single universal workstation such as a console. The object is not
to control every task for every person from one place. A given individual should be able to
accomplish the set of tasks pertinent to that individual’s job from one place.

Ideally, you can have multiple consoles, each tailored to a particular set of tasks; for each
such console, you can have either a duplicate of that console, or some other mechanism to
ensure that for every task, there is an alternative way to accomplish the task in the event the
console or its connection to the sysplex fails.

IBM and non-IBM products are furthering the ability to implement single point of control
through integrating operations on a workstation. IBM provides an implementation of an
integrated operations console through the Tivoli® Management Environment (TME) 10.

Chapter 1. Base and Parallel Sysplex 9

1.6 Parallel Sysplex workload balancing

MVS A MVS C
99% BUSY 10% BUSY

MVS D
60% BUSY
MVS B
100% BUSY
MVS E
40% BUSY

MVS A MVS C
80% BUSY 83% BUSY

MVS D
82% BUSY
MVS B
85% BUSY
MVS E
78% BUSY

Figure 1-6 Parallel Sysplex workload balancing

Parallel Sysplex workload balancing

You might be wondering what a sysplex could do for you. If your data center is responsible for
even one of the following types of work, you could benefit from a sysplex:
򐂰 Large business workloads that involve hundreds of end users, or deal with volumes of
work that can be counted in millions of transactions per day.
򐂰 Work that consists of small work units, such as online transactions, or large work units that
can be subdivided into smaller work units, such as queries.
򐂰 Concurrent applications on different systems that need to directly access and update a
single database without jeopardizing data integrity and security.

Sharing the workload

A sysplex shares the processing of work across z/OS systems and as a result offers benefits
such as reduced cost through more cost-effective processor technology using IBM software
licensing charges in Parallel Sysplex.

Workload balancing
When you are in an environment with multiple systems, the set of performance issues
changes. Existing mechanisms for managing system performance are complex and
single-system oriented.

10 ABCs of z/OS System Programming: Volume 5

To reduce the complexity of managing a sysplex, MVS workload management provides
dynamic sysplex-wide management of system resources. MVS workload management is the
combined cooperation of various subsystems (such as CICS, IMS, and VTAM) with the MVS
workload manager (WLM) component. An installation defines performance goals and a
business importance to workloads through WLM. Workload management focuses on
attaining these goals through dynamic resource distribution.

Workload management
A sysplex provides a different way of managing workloads than was previously used. The
emphasis is on defining performance goals for work, and having MVS and the subsystems
adapt to meet the goals. This provides for the following:
򐂰 A platform for continuous availability so that applications can be available 24 hours a day,
7 days a week, 365 days a year (or close to it).
򐂰 The ability to do more work to provide greater capacity and an improved ability to manage
response time through the use of goals.
򐂰 Greater flexibility and the ability to mix levels of hardware and software.
򐂰 The ability to dynamically add systems, which allows an easy path for incremental growth.
򐂰 Considering resource sharing (of unit tapes, for example), resources are used in the
system that needs them, instead of being dedicated. That enables better environment
management, performance, and cost savings.
򐂰 Data sharing and resource sharing are the fundamental mechanisms that allow
installations to have their workload dynamically redistributed between images in the
Parallel Sysplex. Workload can be routed to systems where spare capacity exists,
avoiding CEC upgrade and still meeting service level objectives, as shown in Figure 1-6.
For example, CICS is usually implementing the function shipping capability toward the file
owner region (FOR) in order to share VSAM files across multiple CICS application
regions. Most of the time this FOR region becomes a bottleneck and a single point of
failure. In a Parallel Sysplex environment, with data sharing, transactions can be routed to
a CICS address space with workload balancing and better response time in the Parallel
Sysplex. This requires a transaction management tool, like CICSPlex SM, but the basic
framework is the Parallel Sysplex.

Sysplex configurations
A sysplex configuration can be either a base sysplex or a Parallel Sysplex. Later in this
chapter, after introducing all the hardware and software required in a sysplex, both
configurations are described. The configurations are described in “Base sysplex” on page 51
and “Parallel Sysplex” on page 52.

Chapter 1. Base and Parallel Sysplex 11

1.7 Sysplex software

System software

Networking software

Data management software

Transaction management software

Systems management software

Figure 1-7 Sysplex software

Sysplex software
The following types of software exploit sysplex capabilities:
System System software is the base software that is enhanced to
support a sysplex. It includes the z/OS operating system, JES2
and JES3, and DFSMS.
Networking Includes Virtual Telecommunications Access Method (VTAM)
and Transmission Control Protocol/ Internet Protocol (TCP/IP),
which support attachment of a sysplex to a network.
Data management Data management software includes data managers that
support data sharing in a sysplex, such as Information
Management System Database Manager (IMS DB), DATA
BASE 2 (DB2), and Virtual Storage Access Method (VSAM).
We can also include here Adabas and Oracle.
Transaction management Transaction management software includes transaction
managers that support a sysplex such as Customer
Information Control System (CICS Transaction Server),
Information Management System Transaction Manager
(IMS TM) and WebSphere Application Services.
Systems management Systems management software includes a number of software
products that are enhanced to run in a sysplex and exploit its
capabilities. The products manage accounting, workload,
operations (as DFSMShsm), performance (as RMF), security
(as RACF), and configuration (as HCD).

12 ABCs of z/OS System Programming: Volume 5

1.8 Sysplex hardware

z10 EC
Coupling Facility

CF Links
z196
12
11 1
10 2
9 3
8 4
7 5
6

Sysplex
Timer
ESCON|FICON
z10 BC

Shared Data
Figure 1-8 Sysplex hardware

Sysplex hardware
A sysplex is a collection of MVS systems that cooperate, using certain hardware and software
products, to process work. A conventional large computer system also uses hardware and
software products that cooperate to process work. A major difference between a sysplex and
a conventional large computer system is the improved growth potential and level of availability
in a sysplex. The sysplex increases the number of processing units and MVS operating
systems that can cooperate, which in turn increases the amount of work that can be
processed. To facilitate this cooperation, new products were created and old products were
enhanced. The following types of hardware participate in a sysplex.

System z processors
Selected models of System z processors can take advantage of a sysplex. These include
large water-cooled processors, air-cooled processors, and the processors that take
advantage of (CMOS) technology.

Coupling facility
Coupling facilities enable high performance multisystem data sharing. Coupling facility links,
called channels, provide high speed connectivity between the coupling facility and the central
processor complexes that use it. These z/OS systems can be located in the same or in
different CPCs that have the coupling facility.

Chapter 1. Base and Parallel Sysplex 13

 Sysplex Timer
The Sysplex Timer® is an external time reference (ETR) device that synchronizes the
time-of-day (TOD) clocks across multiple CPCs in a sysplex. The time stamp from the
Sysplex Timer is a way to monitor and sequence events within the sysplex. Server Time
Protocol (STP) is a server-wide facility providing capability for multiple z/OSs to maintain TOD
time synchronization with each other and form a Coordinated Timing Network (CTN). It is a
cost saving replacement for the Sysplex Timer.

FICON and ESCON

The term Fibre Connection (FICON®) represents the architecture as defined by the
InterNational Committee of Information Technology Standards (INCITS), and published as
ANSI standards. FICON also represents the names of the various System z server I/O
features. ESCON® and FICON control units and I/O devices provide the increased
connectivity necessary among a greater number of systems.

ESCON channels and directors, and FICON channels and directors (switches), are
Enterprise Systems Connection (ESCON) and Fiber Connection (FICON) channels that
enhance data access and communication in the sysplex. The ESCON directors and FICON
switches add dynamic switching capability for those channels.

FICON is widely used in the System z environment, and provides additional strengths and
capabilities compared to the ESCON technology. Many additional capabilities have been
included in support of FICON since it was originally introduced. Some control units and
control unit functions might require FICON use exclusively. For example, Hyper Parallel
Access Volume requires the use of FICON and will not work with ESCON.

14 ABCs of z/OS System Programming: Volume 5

1.9 Sysplex Timer

Coupling Facility Coupling Facility

Sysplex Timer

System z Server 9
10
11 12 1
2
3 9
10
11 12 1
2
3
System z Server
8 4 8 4
7 6 5 7 6 5

FICON Director 1 FICON Director n

DASD DASD

Figure 1-9 Sysplex Timer

Sysplex Timer
Time is a key variable for commercial programs. In physics there are three time scales:
򐂰 Universal international time (UIT) based on the earth revolution cycles. Not used for
modern purposes due to the variability of such cycles.
򐂰 International atomic time (TAI) based on the radioactive properties of Cesium 133.
򐂰 Coordinated universal time (UTC) is derived from TAI, but kept not far from UIT by adding
or deleting discrete units of leap seconds.

Several System z processors can execute several task programs in a data processing
complex. Each of these processors has a time-of-day (TOD) clock, which is an internal
104-bit register incremented by adding a one in bit position 51 every microsecond. TOD
clocks use a UTC time scale.

Multiple TOD clocks

When tasks are shared among different CPUs, multiple TOD clocks can be involved. These
clocks might be in sync with one another. All CPU TODs from the same CPC are internally
synchronized. Then, there is a need for a single time resource, that is, an External Time
Reference (ETR) to synchronize the TOD clocks of CPUs located in distinct CPCs running
z/OS in the same sysplex. The Sysplex Timer is hardware that is used when the sysplex
consists of z/OS systems running in more than one CPC.

Chapter 1. Base and Parallel Sysplex 15

 Note: Currently, you have the possibility of replacing a Sysplex Timer by Server Time
Protocol (STP), as shown in “Server Timer Protocol (STP)” on page 17. The Sysplex Timer
provides synchronization for the TOD clocks of multiple CPUs in distinct CPCs, and
thereby allows events started by different CPUs to be properly sequenced in time. For
example, when multiple CPUs update the same database, all updates are required to be
time stamped in proper sequence in the database log.

Timer functions
There is a long-standing requirement for accurate time and date information in data
processing. As single operating systems have been replaced by multiple, coupled operating
systems on multiple servers, this need has evolved into a requirement for both accurate and
consistent clocks among these systems. Clocks are said to be consistent when the difference
or offset between them is sufficiently small. An accurate clock is consistent with a standard
time source.

The IBM z/Architecture, Server Time Protocol (STP), and External Time Reference (ETR)
architecture facilitates the synchronization of server time-of-day clocks to ensure consistent
time stamp data across multiple servers and operating systems. The STP or ETR architecture
provides a means of synchronizing TOD clocks in different servers with a centralized time
reference, which in turn might be set accurately on the basis of an international time standard
(External Time Source). The architecture defines a time-signal protocol and a distribution
network, which permits accurate setting, maintenance, and consistency of TOD clocks.

External time reference (ETR)

External time reference hardware facility (ETR) is the generic name for IBM Sysplex Timer.
The ETR architecture provides a means of synchronizing TOD clocks in different CPCs with a
centralized time reference, which in turn can be set accurately on the basis of UTC time
standard (External Time Source). The architecture defines a time-signal protocol and a
distribution network (called the ETR network) that permits accurate setting, maintenance, and
consistency of TOD clocks.

Timing services
Timing services are implemented in z/OS by a time supervisor component. It can be used by
an application program to obtain the present date and time, and convert date and time
information to various formats. Interval timing lets your program set a time interval to be used
in the program logic, specify how much time is left in the interval, or cancel the interval. For
programs that are dependent upon synchronized TOD clocks in a multi CPC environment,
like a database, it is important that the clocks are in ETR synchronization. These programs
can use the STCKSYNC macro to obtain the TOD clock contents and determine if the clock is
synchronized with an ETR. STCKSYNC also provides an optional parameter, ETRID, that
returns the ID of the ETR source with which the TOD clock is currently synchronized.

ETR attachments
The ETR feature in System z9® and System z10® servers provides the interface to a
Network Time Protocol (NTP) server with pulse per second (PPS) support.

Note: The zEnterprise 196 does not support the IBM Sysplex Timer and ETR attachment.

16 ABCs of z/OS System Programming: Volume 5

1.10 Server Timer Protocol (STP)

ETS HMC

z10 EC
Preferred
z10 EC
Time Server Backup
Stratum 1 Time Server
Stratum 2

P1 P2

z10 BC
Backup
arbiter Time
Server
Stratum 2
P3

STP can use existing coupling

links

Figure 1-10 Server Time Protocol (STP)

Server Time Protocol (STP)

Server Time Protocol (STP) is designed to help multiple System z servers maintain time
synchronization with each other, without the use of a Sysplex Timer. STP uses a
message-based protocol in which timekeeping information is passed over externally defined
coupling links, such as InterSystem Channel-3 (ISC-3) links configured in peer mode,
Integrated Cluster Bus-3 (ICB-3) links, and Integrated Cluster Bus-4 (ICB-4) links. These can
be the same links that already are being used in a Parallel Sysplex for coupling facility
message communication.

STP is implemented in the Licensed Internal Code (LIC) of System z servers and CFs for
presenting a single view of time to PR/SM™.

Note: A time synchronization mechanism, either IBM Sysplex Timer or Server Time
Protocol (STP), is a mandatory hardware requirement for a Parallel Sysplex environment
consisting of more than one server.

STP link
An STP link is a coupling facility connection that serves as a timing-only link. With STP links,
you can allow multiple servers to form a Coordinated Timing Network (CTN), which is a
collection of servers and coupling facilities that are synchronized to a time value called
Coordinated Server Time. Establishing an STP link between two processors does not require
a CF partition; an STP link can be established between two OS partitions. For an STP link,

Chapter 1. Base and Parallel Sysplex 17

 HCD generates a control unit of type “STP” on both sides of the connection. No devices are
defined.

This server-wide facility provides capability for multiple z/OS systems to maintain TOD
synchronization with each other and form a Coordinated Timing Network (CTN), that is, a
collection of z/OS systems that are time synchronized to a time value. In Figure 1-10, the
preferred time is as follows:
򐂰 P1 is the z/OS preferred timer server (stratum 1) that synchronizes the TODs of the z/OS
systems (stratum 2).
򐂰 P2 is the backup time server, ready to replace P1.
򐂰 P3 is the arbiter time server that decides when the replacement should be done.

The External Time Reference connections are replaced by the implementation of STP, which
makes use of coupling links to pass timing messages to the servers. Transition to STP makes
it possible to have a Mixed Coordinated Network configuration. The Sysplex Timer provides
the timekeeping information in a Mixed CTN. Once an STP-only configuration is established,
the ETR connections are no longer needed. STP allows coexistence with Sysplex Timer in
mixed configurations. The Sysplex Timer console is replaced by an HMC screen for each
possible time zone.

Note: A z196 cannot be connected to a Sysplex Timer; consider migrating to an STP-only

Coordinated Time Network (CTN) for existing environments. It is possible to have a z196
as a Stratum 2 or Stratum 3 server in a Mixed CTN, as long as there are at least two
System z10 or System z9 servers attached to the Sysplex Timer operating as Stratum 1
servers.

Defining STP links in a sysplex

Server Time Protocol is designed to help multiple System z servers maintain time
synchronization with each other, without the use of a Sysplex Timer. STP uses a
message-based protocol in which timekeeping information is passed over externally defined
coupling links, such as InterSystem Channel-3 (ISC-3) links configured in peer mode,
Integrated Cluster Bus-3 (ICB-3) links, and Integrated Cluster Bus-4 (ICB-4) links. These can
be the same links that already are being used in a Parallel Sysplex for coupling facility
message communication.

An STP link is a coupling facility connection that serves as a timing-only link. With STP links,
you can allow multiple servers to form a Coordinated Timing Network (CTN), which is a
collection of servers and coupling facilities that are synchronized to a time value called
Coordinated Server Time. Establishing an STP link between two processors does not require
a CF partition; an STP link can be established between two OS partitions. For an STP link,
HCD generates a control unit of type STP on each side of the connection. No devices are
defined.

You can establish an STP link between two System z servers (z890, z990, z9 EC, or later). In
the Connect to CF Channel Path dialog, select two CHPIDs defined for coupling facilities, and
then specify the Timing-only link option to create an STP link.

18 ABCs of z/OS System Programming: Volume 5

1.11 Coupling facility

Coupling Facility Coupling Facility

Sysplex Timer

11 12 1 11 12 1
10 2 10 2
9 3 9 3
8 4 8 4
7 6 5 7 6 5

System z Server System z Server

Figure 1-11 Coupling facility

Coupling facility
A coupling facility is a special logical partition that runs the coupling facility control code
(CFCC) and provides high-speed caching, list processing, and locking functions in a sysplex.
HCD enables you to specify whether a logical partition can be a coupling facility, operating
system, or either on certain processors. You connect the coupling facility logical partition to a
processor through the coupling facility channels.

With z/OS services, a component called XES allows authorized applications, such as
subsystems and z/OS components, to use the coupling facility to cache data, exchange
status, and access sysplex lock structures in order to implement high performance data
sharing and rapid recovery from failures.

Coupling facility control code (CFCC)

IBM CFCC is licensed internal code (LIC) and always runs under an LPAR, regardless of
whether the CF is in a standalone CPC or in a general purpose CPC (where CFCC LPs are
together with z/OS LPs). A standalone CPC is a CPC only allowing LPs running CFCCs.

CFCC is a simple but efficient operating system where, for example, no virtual storage
support is implemented. It has multiprocessing capabilities running multiple processors and
when there is no work to do, it loops in the CF link waiting for work requests (the interrupt
mechanism is not implemented).

Chapter 1. Base and Parallel Sysplex 19

 A coupling facility (CF) runs the coupling facility control code (CFCC) that is loaded into main
storage at power-on reset (POR) time. CFCC can run on a stand-alone CF server or in a
logical partition.

Coupling facility logical partition (LP)

The coupling facility LP is defined through HCD and processor resource/systems manager
(PR/SM) panels on the Hardware Management Console (HMC). Once you have defined an
LP to be a coupling facility LP, only the CFCC can run in that LP. When you activate the
coupling facility LP, the system automatically loads the CFCC from the laptop support element
(SE) hard disk of the CPC. Its major functions are:
򐂰 Storage management
򐂰 Support for CF links
򐂰 Console services (HMC)
򐂰 Trace, logout, and recovery functions
򐂰 Provide support for the list, cache, and lock structures

20 ABCs of z/OS System Programming: Volume 5

1.12 Message time ordering

z9
z/OS

Sysplex
LPARs CF01
z196 ICF
z10
B
IF Required connection IS IC
PS er
) C- B-
4
e from CF on System z 3
(p (p
-3 to Sysplex Timer eer
C )
IS
z/OS Sysplex
IC PSIFB LPARs
Sysplex
CF02 ISC-3 (peer) z/OS
LPARs
ICF

11 12 1 11 12 1
10 2 10 2
9 3 9 3
8 4 8 4
7 6 5 7 6 5

Sysplex Timer
FICON / ESCON

Disks Disks Disks

Figure 1-12 Message time ordering

Parallel Sysplex configurations

As server and coupling link technologies have improved over the years, the synchronization
tolerance between operating system images in a Parallel Sysplex has become more rigorous.
In order to ensure that any exchanges of time stamped information between operating system
images in a Parallel Sysplex involving the CF observe the correct time ordering, time stamps
are included in the message transfer protocol between the server operating system images
and the CF. This is known as message time ordering.

Message time ordering

Figure 1-12 illustrates the components of a Parallel Sysplex as implemented within the
zSeries architecture. Shown is an ICF connection between two z10 EC servers running in a
sysplex, and there is a second integrated coupling facility defined within one of the z10s
containing sysplex LPARs. Shown also is the connection required between the coupling
facility defined on a z10 and the sysplex timer to support message time ordering. Message
time ordering requires a CF connection to the Sysplex Timer.

Before listing the message time ordering facility rules, a short description of the message time
ordering facility is included here. When the CF receives the message, it verifies that the
message’s time stamp is less than the CF’s TOD clock. If the time stamp in the message is
ahead of the CF’s TOD clock, the message is not processed until the CF TOD clock catches
up to the message time stamp value.

Chapter 1. Base and Parallel Sysplex 21

 Mixed or STP-only Parallel Sysplex
In a mixed or STP-only CTN in a Parallel Sysplex configuration, the requirement is that all
servers support the message time ordering facility.

The following message time ordering facility rules are enforced when there is a mixed CTN in
a Parallel Sysplex configuration:
򐂰 z/OS images running on STP-configured servers can connect to CFs that are on servers
that are not STP capable only if the coupling facility supports message time ordering
facility and is attached to the Sysplex Timer.
򐂰 CFs on an STP-configured server can connect to z/OS images running on servers that are
not STP capable only if the non-STP-capable server supports message time ordering
facility and is attached to the Sysplex Timer.

For more details about this topic, see the document Server Time Protocol Planning Guide,
SG24-7280.

22 ABCs of z/OS System Programming: Volume 5

1.13 Coupling facility LPARs and CFCC code

Standalone coupling facility

Internal coupling facility (ICF)

Coupling facility configuration options

CF storage

CF control code level (CFLEVEL)

Figure 1-13 Coupling facility LPARs and CFCC code

Standalone coupling facility (CF)

A standalone CF is a CPC where all the processor units (PUs), links, and memory are for
CFCC use. This means that all LPs running in such a CPC run only CFCC code. The
standalone CF is also called external coupling facility.

Internal coupling facility (ICF)

ICFs are PUs in a CPC configured to run only CFCC code. The PUs are not shipped with a
fixed function assignment (personality), but are assigned during power-on reset (POR) or
later non-disruptively by on demand offerings such as: CBU, CuOD, CIU, ON/OFF COD.
Those offerings allow the customer to convert, in seconds, a non-characterizable PU in any
PU personality type such as: CPU, ICF, IFL, zAAP, zIIP and SAP.

An ICF can reduce the cost of exploiting coupling facility technology because:
򐂰 ICFs are less expensive than CPs.
򐂰 An ICF has a special software license charge. Special PR/SM microcode prevents the
defined ICF PUs from executing non-CFCC code such as z/OS.

Coupling facility configuration options

A coupling facility always runs CFCC code within a PR/SM LPAR license internal code (LIC).
As we already saw, a CF LPAR can be configured in one of two ways:
򐂰 In a standalone CPC, only coupling facilities are present

Chapter 1. Base and Parallel Sysplex 23

 򐂰 In a general purpose CPC, the CF LP can co-exist with LPs running z/OS code, or even
Linux® (under z/VM® or not). The z/OS LPs can be either in the same Parallel Sysplex as
the coupling facility or not.

Decisions regarding where to configure CF LPs are based mostly on price/performance,

configuration characteristics, CF link options, and recovery characteristics (availability).

CF storage
CFCC formats central storage in contiguous pieces called structures. Structures can be used
to keep data by software exploiters (authorized programs) such as: z/OS components,
subsystems, products. The exploiters may have several instances running in different z/OS
systems in the sysplex. The major reason for having structures is to implement data sharing
(with total integrity), although the structures may be used as high-speed caching memory.
Structure types are:
򐂰 Lock structure
򐂰 List structure
򐂰 Cache structure

Each structure type provides a specific function to the exploiter. Some storage in the coupling
facility can also be allocated as a dedicated dump space for capturing structure information
for diagnostic purposes. In order to access the coupling facility structures, z/OS systems
(running the exploiters) must have connectivity to the coupling facility through coupling
facility links. Refer to “Coupling facility links” on page 25 for more information.

Coupling facility control code level (CFLEVEL)

The level (CFLEVEL) of the coupling facility control code (CFCC) that is loaded into the
coupling facility LPAR determines what functions are available for exploiting applications.
Various levels provide new functions and enhancements that an application might require for
its operation. As more functions are added to the CFCC, it might be necessary to allocate
additional storage to a coupling facility structure. Similarly, as new functions are added, the
coupling facility itself may require additional storage. In any configuration, the amount of fixed
storage required for the coupling facility is based on configuration-dependent factors.

To implement a coupling facility in your sysplex requires both hardware and software, as
follows:
򐂰 CPC that supports the CFCC.
򐂰 CPCs on which one or more z/OS images run and which are capable of connecting to the
coupling facility with CF links.
򐂰 Appropriate level of z/OS that allows an exploiter to access a desired function when
managing the coupling facility resources.
򐂰 CFCC must implement the functions the exploiter needs.

To support migration from one CFCC level to the next, you can run several levels of CFCC
concurrently as long as the coupling facility logical partitions are running on different servers
(CF logical partitions running on the same server share the same CFCC level).

24 ABCs of z/OS System Programming: Volume 5

1.14 Coupling facility links

z9
z/OS

Sysplex
LPARs CF01
z196 ICF
z10
B
IF Required connection ISC IC
PS er
) B-
4
e from CF on System z -3
(p (p
3 to Sysplex Timer e er
C- )
IS
z/OS Sysplex
IC PSIFB LPARs
Sysplex
CF02 ISC-3 (peer) z/OS
LPARs
ICF

11 12 1 11 12 1
10 2 10 2
9 3 9 3
8 4 8 4
7 6 5 7 6 5

Sysplex Timer
FICON / ESCON

Disks Disks Disks

Figure 1-14 Coupling facility links

Coupling facility links

To enable the communication between a coupling facility (CF) logical partition (LP) and the
z/OS (LPs), special types of high-speed CF links are required. These links are important
because of the impact of link performance on CF request response times. For configurations
covering large distances, time spent on the link can be the largest part of CF response time.

The coupling link types for the coupling facility are:

(PSIFB, IC, ICB, and ISC-3)

A CF link adapter can be shared between LPs, meaning the same adapter can transfer data
from/to different z/OS systems to one CF, thus reducing the number of links needed. This is
called multiple image facility (MIF), the same name used for FICON and ESON channels.

CF links in the System z servers work in a mode called peer mode. In this mode we have even
more flexibility with connections. For example, a single link adapter can be connected
(multiple image facility) to both z/OS and a CF.

Both the coupling facility LPs and the CF links must be defined to the I/O configuration data
set (IOCDS). Hardware configuration definition (HCD) provides the interface to accomplish
these definitions and also automatically supplies the required channel control unit and I/O
device definitions for the coupling facility channels.

Chapter 1. Base and Parallel Sysplex 25

 Internal coupling (IC) link
The IC is a zSeries or z9 connectivity option that enables high-speed connection (more than 3
GB/sec) between a CF LP and one or more z/OS LPs running on the same zSeries or z9
CPC. The IC is a linkless connection (implemented in Licensed Internal Code) and so does
not require any hardware or cabling.

InterSystem Channel (ISC) link

InterSystem Channel provides connectivity through optical cables. ISC links are point-to-point
connections that require a unique channel definition at each end of the link. There are two
types of ISC features supported, ISC-3 and ISC-2. In modern CPCs such as System z
servers (z800, z900, z890, z990 and z9), there is support for ISC-3 only. They are used for
distances beyond 7 meters and allow a rate of up to 200 MBps.

Integrated Cluster Bus (ICB)

Integrated Cluster Bus provides connectivity through copper cables. They are faster than ISC
links, attaching directly to a Self-Timed Interconnect (STI) bus of the CEC cage. They are the
preferred method for coupling connectivity when connecting System z servers over short
distances (up to 7 meters). For longer distances, ISC-3 links must be used. There are two
types of ICB links available:
򐂰 ICB-4 links provide 2 GB/sec coupling communication between z990, z890, and z9 CPCs.
򐂰 ICB-3 links provide 1 GB/sec coupling communication between z800, z900, z990, z890,
and z9 CPCs.

InfiniBand coupling links (PSIFB)

InfiniBand coupling links (PSIFB) are high speed links on z196, z10, and z9 servers. The
PSIFB coupling links originate from three types of fanout. PSIFB coupling links of either type
are defined as CHPID type CIB in HCD/IOCP, as follows:
򐂰 HCA2-O (FC 0163)
The HCA2-O fanouts support InfiniBand Double Data Rate (IB-DDR) and InfiniBand
Single Data Rate (IB-SDR) optical links. The HCA2-O fanout supports PSIFB coupling
links at distances of up to 150 meters. PSIFB coupling links operate at 6 GBps
(12x IB-DDR) when connecting a z196 or z10 to z196 and z10 servers, and at 3 GBps
(12x IB-SDR) when connecting a z196 or z10 to a z9. The link speed is auto-negotiated to
the highest common rate.
򐂰 HCA2-O LR (FC 0168)
The HCA2-O LR fanout supports PSIFB Long Reach (PSIFB LR) coupling links for
distances of up to 10 km and up to 100 km when repeated through a DWDM. This fanout
is supported on z196 and z10. PSIFB LR coupling links operate at up to 5.0 Gbps (1x
IB-DDR) between z196 and z10 servers, or automatically scale down to 2.5 Gbps (1x
IB-SDR) depending on the capability of the attached equipment.
򐂰 HCA1-O (FC 0167)
The HCA1-O fanout supports InfiniBand Single Data Rate (IB-SDR) optical links. The
HCA1-O fanout supports PSIFB coupling links at distances of up to 150 meters. PSIFB
coupling links operate at 3 GBps (12x IB-SDR) when connecting the z9 server to a z196 or
z10 server.

Publications of interest
zSeries Connectivity Handbook, SG24-5444, Processor Resource/Systems Manager
Planning Guide, SB10-7036, z/OS MVS Setting Up a Sysplex, SA22-7625, and HCD
Planning, GA22-7525.

26 ABCs of z/OS System Programming: Volume 5

1.15 Sysplex overview

Cross-system coupling facility (XCF)

Sysplex couple data sets
Sysplex configurations
Base sysplex
Parallel Sysplex
XCF and XES
Sysplex couple data sets
Coupling facility structures
PARMLIB members requirements
IEASYSxx
CLOCKxx
COUPLExx
Multisystem consoles
Sysplex related z/OS commands

Figure 1-15 Sysplex overview

Sysplex overview
Now that the pieces that make up a sysplex have been introduced, the remainder of this
chapter presents an overview of the sysplex. Figure 1-15 is an overview of the remaining
topics to be discussed in this chapter.

It is difficult to explain the sysplex without first explaining the cross-system coupling facility
(XCF) and its services, so we start by describing XCF, its services and exploiters. Take note
that despite having in the name the expression “coupling facility,” XCF is not able to access
such coupling facility directly.

Next, the sysplex configurations and how system consoles are used to enter sysplex-related
commands are described, as follows:
򐂰 Fundamentals of a base sysplex (a function of XCF), what is required and how to define it.
򐂰 An overview of Parallel Sysplex and how to migrate from a base to a Parallel Sysplex.
򐂰 Which PARMLIB members you have to change in order to define a sysplex and which
changes are necessary.
򐂰 How to use consoles in a sysplex environment; how many consoles you need and which
ones are mandatory.
򐂰 The sysplex-related commands, how to direct a command to a specific console, and how
to reply to messages.

Chapter 1. Base and Parallel Sysplex 27

1.16 Cross-system coupling facility (XCF)

CTC Links
CICS1
CICSD CICSA
X X
CICS2
CICSE CICSB
C C
F F
CICS3
CICSF CICSC

MVS1 MVS2

Inter-region communication through XCF

Inter-system communication through XCF using XCF links

Figure 1-16 Cross-system coupling facility (XCF)

Cross-system coupling facility (XCF)

The cross system coupling facility (XCF) component of z/OS provides simplified multisystem
management. XCF services allow authorized programs on one system to communicate with
programs on the same system or on other systems. If a system fails, XCF services also
provide the capability for batch jobs and started tasks to be restarted on another eligible
system in the sysplex.

XCF groups
An XCF group is a set of related members that a multisystem application defines to XCF. A
member is a specific function, or instance, of the application. A member resides on one
system and can communicate with other members of the same group across the sysplex.

Communication between group members on different systems occurs over the signaling
paths that connect the systems; on the same system, communication between group
members occurs through local signaling services. To prevent multisystem applications from
interfering with one another, each XCF group name in the sysplex must be unique.

CICS address spaces

Each CICS address space in Figure 1-16 is an XCF instance. The terms instance, exploiting
instance, or exploiter are used to denote the active subsystem or address space of a
component, product, or function that directly exploits sysplex services. XCF is a z/OS
component that provides services to allow multiple instances (named members) of an
application or subsystem, running on different systems in a sysplex, to share status

28 ABCs of z/OS System Programming: Volume 5

information and communicate through messages with each other. A set of same instance
members is called a group. A member of an application can use XCF services to:
򐂰 Inform other members of their status (active, failed, and so forth).
򐂰 Obtain information about the status of other members, such as whether another member
failed.
򐂰 Send messages to and receive messages from each other member, in the same or in
another z/OS. The most frequent case is other z/OSs (inter-system communication).

Automatic Restart Manager (ARM)

If z/OS fails, XCF can call Automatic Restart Manager (ARM) services (a z/OS component) to
provide the capability for batch jobs and started tasks to be restarted on another eligible z/OS
in the sysplex. Then an application is allowed to:
򐂰 Request automatic restart in the event of application or system failure
򐂰 Wait for another job to restart before restarting
򐂰 Indicate its readiness to accept work
򐂰 Request that automatic restart no longer be performed
򐂰 Indicate that automatic restart should be performed only if the backup copy of the
application data no longer exists

Chapter 1. Base and Parallel Sysplex 29

1.17 Base sysplex

Cross System Coupling Facility

ETR
12
11 1

SC47 SC52
10 2
9 3
8 4
7 5
6

GROUP SYSGRS SYSGRS GROUP

Member SC47 X X SC52 Member

CTC
GROUP C C GROUP
Member Member

F F

Figure 1-17 Base sysplex

Base sysplex
A base sysplex configuration is a sysplex with no coupling facilities. The base sysplex can be
composed of one or more z/OS systems that have an XCF sysplex name and in which the
authorized programs (members) use XCF services. XCF services are available in both single
and multisystem environments. A multisystem environment is defined as two or more z/OS
systems residing on one or more CPCs’ logical partitions connected through CTCs.

A base sysplex is the first step to implementing a Parallel Sysplex. A Parallel Sysplex is a
base sysplex plus the use of the coupling facility. So, when you introduce the coupling facility,
XCF exploits the coupling facility, using it as a link between z/OS systems.

Figure 1-17 shows how XCF works in a multisystem sysplex. Each z/OS has an XCF
component that handles groups for the participating members. Here you can see as an
example the group named SYSGRS, which is very important for global resource serialization
(in the group the name of each member is the name of the z/OS system). This group has one
member (GRS component of z/OS) in each z/OS. Other groups can be: consoles (SYSMCS),
JES2, JES3, WLM, and others. These groups are described in detail in 1.23, “XCF exploiters”
on page 39.

The communication link between XCFs could be through channel-to-channel adapters

(CTCs) that allow data movement between XCF buffers in the systems through an I/O
operation. Another option for linking is a CF list structure, which is discussed later in this
chapter.

30 ABCs of z/OS System Programming: Volume 5

1.18 XCF application, member, and group

SYSPLEX PLEX1

MVS A MVS B MVS C

DB2G4
DB2G1 DB2G2 DB2G3 Group
Group Group Group DB2V71G
DB2V71G DB2V71G DB2V71G Member 4-
Member 2 Member 3 Member 1 task xxx
Member 5

Figure 1-18 XCF application, member, and group

XCF application, member, and group

An application in a XCF context is a program that has various functions distributed across
z/OS systems in a multisystem environment. An application or subsystem might consist of
multiple instances, each running on a different system in the same sysplex. Typically, each
instance performs certain functions for the application as a whole. Alternatively, each instance
could perform all the application’s functions on a given system. XCF services is available to
authorized applications, such as subsystems and z/OS components, to use sysplex services.

Member
A member is a specific function (one or more routines) of a multisystem application that is
joined to XCF and assigned to a group by the multisystem application. A member concept
applies to all authorized routines running in the address space that issued the IXCJOIN
macro service. Only for termination purposes (resource clean-up), the member can be
associated with an address space, job step, or task. XCF terminates the member when its
association ends. The same address space can have more than one group.

Group
A group is the set of related members defined to XCF by a multisystem application in which
members of the group can communicate with other members of the same group. A group can
span one or more of the systems in a sysplex and represents a complete logical entity to XCF.

Chapter 1. Base and Parallel Sysplex 31

1.19 XCF services

Group services

Administrating members and groups

Signaling services

Control exchange of messages between members

Status monitoring services

Monitor status of members and notify others

Figure 1-19 XCF services

XCF services
z/OS XCF allows up to 32 z/OS systems to communicate in a sysplex. XCF provides the
services that allow multisystem application functions (programs) on one z/OS system to
communicate (send and receive data) with functions on the same or other z/OS systems. The
communication services are provided through authorized assembler macros and are as
follows:
򐂰 Group services
򐂰 Signalling services
򐂰 Status monitoring services

Group services
XCF group services provide ways for defining members to XCF, establishing them as part of a
group, and allowing them to find out about the other members in the group. A member
introduces itself to XCF through the IXCJOIN macro. If a member identifies a group exit
routine, XCF uses this routine to notify this member about status changes that occur to other
members of the group, or systems in the sysplex; thus, members can have the most current
information about the other members in their group without having to query each other.

Signaling services
The signaling services control the exchange of messages between members of an XCF
group. The sender of a message requests services from XCF signaling services. XCF uses

32 ABCs of z/OS System Programming: Volume 5

buffer pools to communicate between members in the same system, and it uses buffer pools
plus signaling paths (CTCs or a CF list structure) to send messages between systems in the
sysplex.

Status monitoring services

Status monitoring services provide a way for members to determine their own operational
status and to notify the other members of the group when that operational status changes. An
installation-written status exit routine identified to XCF determines whether a member is
operating normally. An installation-written group exit routine identified to XCF allows a
member to maintain current information about other members in the group, and systems in
the sysplex.

Chapter 1. Base and Parallel Sysplex 33

1.20 XCF signaling paths

CTC device
Coupling facility list structure
Defined in COUPLExx parmlib member
Signaling paths can be:
Default paths - for use by all groups
Dedicated paths - for a specific group through
transport classes
CTC paths are one directional
Requires a PATHIN and a PATHOUT

Figure 1-20 XCF signaling paths

XCF signaling paths

Whether you implement signaling through CTC connections or coupling facility list structures,
you need to specify information in the COUPLExx parmlib member. After an IPL, you can
issue the SETXCF START,PATHOUT or SETXCF START,PATHIN commands to specify outbound or
inbound paths. To establish signaling through a coupling facility, you also need to define a
coupling facility resource management (CFRM) policy.

Then, XCF group members use the signaling mechanism to communicate with each other.
These communication paths can be:
򐂰 Channel-to-channel adapter (CTC) communication connections.
򐂰 Coupling facility through list structures. XCF calls XES services when the path of
communication is a coupling facility.
򐂰 A combination of both, CTC and list structures. In this case XCF selects the faster path.

A message generated in a member has the following path:

򐂰 From the sending member to an XCF buffer in the output buffer pool.
򐂰 From the XCF output buffer through the signaling path:
– If a CTC goes directly to a buffer in the XCF input buffer pool.
– If CF list structure is stored for a while in the structure memory and from it to an XCF
buffer in the input buffer pool. It is like a mail box.

34 ABCs of z/OS System Programming: Volume 5

Random documents with unrelated
content Scribd suggests to you:
of the West Highlands, but owing to the incessant war waged against them
by shepherds, gamekeepers, and vermin-trappers, now become so rare that
we stopped our pony to have a good look at a pair that we saw the other day
near Strontian, at the head of Loch Sunart. If you want a specimen of any
British bird, just commission Mackenzie to get it for you. He will only
bring you a specimen that is perfect of its kind, and if you only give him
time he will succeed in getting it, even if he walked a thousand miles in the
pursuit.

With reference to our explanation of the term study applied to a small

plateau, a well-known spot at the top of Glencoe, a correspondent writes as
follows:—“You do not seem to be aware that study is the word in common
use in Lowland Scotland for an anvil as well as amongst the unlisping
Celts. I wonder you forgot Burns’ well-known lines—

‘Nae mercy, then, for airn or steel;

The brawnie, bainie, ploughman chiel,
Brings hard owrehip, wi’ sturdy wheel
The strong forehammer,
Till block and studdie ring and reel
Wi’ dinsome clamour.’ ”

We are much obliged to our friendly correspondent. The quotation proves

that the Lowland Scotch as well as the Highlanders have a difficulty with
the lisping sound of th, preferring the simpler and more natural sound of d.

A gentleman from Badenoch greatly amused us the other day by his account
of a certain superstitious observance on the part of a “wise woman” in his
neighbourhood. The gentleman’s wife was sitting with her baby, only a few
weeks old, in her lap. It was of course a marvel of a baby; for bigness and
beauty the finest baby, like all babies, that ever was seen, and of which its
parents were naturally and very excusably as proud as proud could be. The
“wise woman” of the place had called to see the child, and congratulated
the parents on their good luck. The crone got a chair opposite to that
occupied by the happy mother, while the father looked on and smiled with
becoming dignity and pride. As the old woman was looking at the child, it
chanced to yawn, bored probably by the amount of attention paid to it, and
getting sleepy. As it yawned, the old woman got up from the chair, and
walking over to the “infant phenomenon,” coolly and deliberately spat in its
face! The mother was horrified; the father in a rage asked what the deuce
she meant by spitting in his son’s face? The old lady quietly answered that
the yawn was owing to an evil influence at that moment at work with the
child, and her spitting in its face was the readiest and most effectual way of
saving it from one or more of the mischievous tricks which ill-natured
fairies are so fond of playing off on babies that are “beautiful exceedingly,”
and more especially when they are overmuch petted and bepraised by their
parents and friends. The “wise woman” was at once liberally supplied with
the refreshments usual on such occasions, and as soon as possible
dismissed, care being taken the while not to offend her, which might have
been a serious matter for baby and all concerned. It is not a little curious
that although in all countries to spit at one is expressive of the utmost
detestation and contempt, yet in the superstitions of the Lowlands of
Scotland, as well as in the Highlands, to spit on a person or thing, under
certain conditions and circumstances, is supposed to be counteractive of
evil influences, and therefore a highly commendable act. We have seen a
woman spit on the nets in a boat as it left the shore, to ensure a successful
fishing; and when hand-line fishing, a man who has had little luck and is
getting impatient, as he baits his hook afresh, spits on it before dropping it
again into the sea, in the belief that good luck attends the act. An old
woman who has just bound up a bruised or broken limb, whether of man or
beast, will sometimes finish the operation by spitting on the bandage. In the
superstitions of most countries, such involuntary and apparently causeless
acts as sneezing and yawning are attributed to supernatural agencies, and
spitting at the sneezer or yawner is still sometimes practised as a counter-
charm by the oldest and most learned professors of such lore, an older
superstition probably than the more common practice of invoking the
Divine blessing on the subjects in such cases. Questionable, therefore, and
rude as at first sight seemed the act, we assured our Badenoch friend that
the “wise woman,” in acting as she did, meant his bairn no evil or
disrespect at all, but the very contrary.
 CHAPTER LVII.

Caught in a Squall on Loch Leven—Potatoes and Herrings: How to cook them—

A day in Glen Nevis—A visit to Uaimh Shomhairle, or Samuel’s Cave—The
Cave-Men.

The reader may remember that we concluded our last with a hopeful and
jubilant note, believing that really fine weather—a long track of it, perhaps
—was just at hand. We much regret having to say that our meteorological
vaticinations proved utterly incorrect. It still rains [July 1877], not
constantly, indeed, but with sufficient persistence to make everybody
miserable, and to reduce our hopes of a good harvest almost to zero.
Yesterday, for example, we had occasion to cross the Loch in our boat. It
was a nice bright day enough at starting, with a fresh breeze from N.W.,
which carried us along at racing pace. All of a sudden the heavens became
black and threatening; a terrible squall almost capsized us ere we had time
to sing out to our companion to let go “everything by the run.” He did,
fortunately, let go just in time, and grasping an oar ourselves, and calling on
him to take another, we had her head turned to the wind and waves as
quietly but as quickly as possible. Thus we held her, just like a horse by the
reins, while the squall lasted, and cunningly took advantage of its drift to
get to the Appin shore. We managed to reach it, but in very sorry plight, as
you shall hear. With the squall had come rain, literally the heaviest we ever
saw, which drenched us to the skin; every drop big enough to fill as it fell
the largest of thimbles, and driven by the squall, remember, it fell with the
force of a spent bullet. As “drookit” and drenched we landed, and crawled
with all the miserable, and woebegone, and shambling gait of the really and
thoroughly through-and-through wet, you would have laughed in the teeth
of all the rain had you only met us; and we much doubt if any one who did
not know us would just then have been disposed to appraise ourselves and
our whole belongings at the value of a much bigger coin of the realm than a
shabby florin. And this is just the sort of weather it continues to be. You
cannot depend upon it for an hour. It is sunshine and blue above just for five
minutes; it is all of a sudden gloomy and black as Erebus, and raining so
multitudinously that you are fain to draw the skirts of your coat anyhow
over your head and run for the nearest shelter. When we are to have better
weather let the meteorologists, who ought to know, say.

There is an old and frequent proverb, though rarely heard now-a-days, to

the effect that “there goes reason to the roasting of eggs,” the meaning of
which, as we apprehend it, is that the smallest culinary operation is of
importance, and should be gone about with judgment and care. If the
proverb, however, in its actual words, as a mere popular saw, is very much
forgotten, it is a good sign of our time that its spirit at least is in this our
own day claiming no little attention, as the establishment of “cookery
classes,” and the praiseworthy attempts to disseminate culinary lore
amongst the people, abundantly testify. It has been said that the man who
makes two blades of grass to grow where only one blade grew before is a
benefactor to his species, and equally so, would we venture to assert, is he a
benefactor to the human race who shows how any single article of food,
usually cooked and served in an unsatisfactory and tasteless fashion, may,
with no extra expense and little extra trouble, be made palatable and
savoury. The other day, landing from our boat, we went into a cottar’s house
close by the sea, in a neighbouring district, just as the gudewife was
preparing the family dinner. A pot of new potatoes was boiling on the fire,
and as she knew that it would take us still some time to get home, she very
good-naturedly invited us to wait a little and take a share with herself and
her husband of the dinner about to be served, a bit of hospitality as frankly
accepted as it was kindly offered. Looking now and again into the boiling
potato pot, and listening with inclined ear to the sound, actually musical in
such a case, of its boil and bubbling, she was ready at the proper instant to
snatch it off the fire, and, carrying it to a corner of the kitchen, she poured
off the water, and immediately re-hung it over the fire again, shortening the
chain by which it was suspended by a link or two, that the fire might not,
now that it was waterless, have too much effect upon it. She then got some
half-dozen fresh herrings, caught early that morning—herrings large, and
beautiful, and silvery scaled as a salmon—and drying them nicely with a
cloth, she placed them flat-wise side by side on the top of the potatoes in
the pot, the lid of which she was careful to make fit tightly by means of a
coarse kitchen towel, which served at once to cover the contents, and to
cause the lid to fit so tightly that all the steam was effectually retained. For
the time being, in short, the pot by this ready expedient may be said to have
been hermetically sealed. During a quarter of an hour, perhaps, and while
the gentleman and ourselves carried on a lively conversation, the wife kept
an attentive eye on the pot, never once lifting the lid, however, but from
time to time raising or lowering a link of the chain as in her judgment was
necessary. All being ready at last, she took the pot off the fire, and set it on a
low stool in the middle of the floor. She then lifted the lid and the cloth, and
the room was instantly filled with a savoury steam that made one’s mouth
water merely to inhale it. Occupying each a low chair, we were invited to
fall to, to eat without knife, or fork, or trencher, just with our fingers out of
the pot as it stood. It was a little startling, but only for a moment. After a
word of grace we dipped our hand into the pot, and took out a potato hot
and mealy, and with the other we took a nip out of the silvery flank of the
herring nearest us. It was a mouthful for a king, sir! We have in our day a
thousand times dined well and heartily both at home and abroad, but we
greatly question if we ever enjoyed a dinner half so much as that. The
savouriness of that potato and herring feast will haunt us till our dying day.
What struck us was simply this: A new potato and fresh herring as usually
served is something terribly insipid; as we got it that day it was a meal for
an emperor. We actually felt inclined to lick our fingers after every
mouthful, than which surely there could be no higher praise of any food
whatever. Let such of our readers as have the opportunity just try a potato
and herring cooked in the manner stated, eating it digitally, with their own
proper fingers, and they will thank us, if they are honest, for bringing so
savoury and delicious a dish to their knowledge.

One of the finest glens in all the West Highlands is Glen Nevis, which,
opening out in the direction of the old Castle of Inverlochy, extends
eastwards and inland, the valley gradually narrowing into glen and gorge as
you proceed, for nine or ten miles, presenting at every turn and standpoint
throughout its many windings a succession of the most striking and
beautiful pictures imaginable, so striking and startling at times, and new at
least in some of their details, that a genuine lover of mountain scenery
wishes that he could devote an entire day to every separate mile of its
extent, rather than have to hurry through it all in something like half a
dozen hours, which is the way the thing is usually done. It is like being
dragged, as happened to us once, by a nervous and impatient lady friend of
ours, at a sort of half trot through a picture gallery, where, if you had your
own way, you would gladly lounge and linger till the custodier of the place,
perhaps, came respectfully to hint that the afternoon was far advanced, and
that shutting-up time was at hand. With the entrance to Glen Nevis, as far as
the mansion-house, we had long been familiar, and once at least we had a
bird’s-eye glance into the glen proper itself, from the summit of
Dundearduil, which we had approached from the south in order to examine
its curious and still inexplicable vitrifications. It was not, however, till
Friday last, that we had an opportunity of thoroughly exploring the glen
through all its windings, and coming with little difficulty to the conclusion
already expressed, that of all our West Highland glens, it is, perhaps, the
most beautiful and (Glencoe always apart) the most deserving of a thorough
and leisurely examination. We were fortunate in having hit upon a highly
favourable day—not too bright, for glaring sunshine and unclouded
brightness amongst mountain scenery is a great mistake—and no less
fortunate in our companions, each one of them blessed with eyes that, open,
could really see, and hearts that, duly appealed to, could truly feel; who
knew full well what they had come to do, and from first to last did it
admirably. Barely, we should say, has the noble glen exposed its stern
grandeur and innumerable beauties under favourable skies, to the glad and
earnest gaze of more intelligently appreciative spectators; and more rarely
still, perhaps, have the splendid falls of the Nevis borne burden to peals of
honester or merrier laughter than we indulged in as over the well-plenished
luncheon basket we fortified ourselves for the ascent of the upper gorges,—
a somewhat “stiff” climb, but neither really difficult nor dangerous. When
we say that at Glen Nevis House our party was joined by Mr. Macpherson
—fear a ghlinne e féin, the goodman of the glen himself, as the Highlanders
say—who kindly accompanied us throughout, and to whom every foot of
the glen was as familiar as the floor of his own dining-room, many of our
readers will understand how really pleasant and enjoyable, cœteris paribus,
must have been our upland wanderings on that delightful day.
We have no intention of entering on anything like a minute or photographic
description of Glen Nevis, for which, indeed, half-a-dozen Nether Lochaber
columns would hardly suffice; we can only hurriedly glance at what most
instantly and indelibly struck us in the day’s excursion. First of all, we were
all struck by the exceeding pellucidity and crystal clearness of the waters of
the Nevis. Nowhere else did we ever see a mountain stream so beautifully
transparent. Standing on the brink of any selected pool, many feet in depth,
you distinguished the smallest pea-sized pebble, its veins, scratches, and
striations, as distinctly as if you had it on the palm of your hand, under a
lens, and within less than a foot focus of your eyeball! And all this
remarkable pellucidity, observe, not in one particular pool, or in any one
particular stretch of the river, but throughout all its beautiful windings.
Another remarkable feature of the glen is the manner in which its natural
birch woods grow. They occupy a pretty broad belt almost half-way up the
mountains, leaving a still broader belt between themselves and the river
banks comparatively bare and treeless. In all the other Highland glens with
which we have any acquaintance, whatever of wood there is always begins,
as seems most natural, at the river banks, where it is thickest and most
luxuriant, growing away and upwards on either side to a greater or less
altitude, according to the nature of the soil and the shelter to be had from
the prevailing winds. And speaking of winds, this is the place to observe
that of all our glens Glen Nevis is perhaps the stormiest, the wind in a gale
not blowing steadily, but in fitful gusts and whirlwind-wise, striking in from
the corries right and left, and meeting in the centre with a force and fury
unimaginable by non-residenters. How do you know, the reader may ask,
for it was calm and quiet enough during your visit on Friday? True, and yet
we failed not to notice a very striking proof of the storminess at times of
Glen Nevis notwithstanding. As you pass the forester’s house at Auchreoch,
lift up your eyes, and please observe how carefully, how thoroughly,
closely, compactly, and painstakingly it is thatched; and observe further and
over all a network of wire as thick and strong as that used in our overland
telegraphy, and to the end of each wire as it almost reaches the ground in
front and at the back of the house, please notice suspended a large stone,
water-worn boulders from the river below, each of a hundredweight or
more, and you will not fail, we think, to understand how we so confidently
decided that Glen Nevis at times must be an exceedingly stormy place. If
you assert that other Highland glens may be quite as stormy in the season of
storms, we shall not contradict you; what we do say is this, that never did a
house-roof speak to us so eloquently of furious and frequent storm and
whirlwind as did the roof of that house at Auchreoch, and a very good
house it is, and a very pretty place to the bargain. A little beyond
Auchreoch, and to the left of the path, there is a bit of wild and rugged rock
scenery well worth attention. Here and there, over the face of what seems
the hard impenetrable rock, many trees grow and flourish as if through the
very heart of the granite. The explanation of course is, that the rock which
seems so homogeneous and solid at a distance is in reality fissured and
fractured in all directions, and that in these fissures the trees find soil and
food enough to sustain a wonderfully luxuriant growth and opulence of
foliage for such a situation. About a mile further up the glen, we separated
from our companions for a while, we having determined to cross the Nevis
at this point in order to visit Uaimh Shomhairle, or Samuel’s Cave, the
entrance to which was pointed out to us by Mr. Macpherson in the face of
the opposite steep. To get across the river we had to strip until in a state of
almost puris naturalibus, and even then it was somewhat dangerous, a
single false step might have been attended by very serious consequences.
With a little circumspection and care, however, we got safely over, and half-
dressed and barefooted we climbed the rock like a chamois, and in less than
ten minutes we were standing at the mouth of the celebrated cave. Samuel’s
Cave is in fact two caves, the outer and smaller one, with a broad portal that
admits abundant light and air, forming a sort of vestibule or antechamber to
the inner cave. Provided with one or two old newspapers and some wax
vestas, we improvised a couple of rude torches which we carried with us as
we crept through a narrow opening by which alone access is obtained into
the inner antrum. Lighting one of these torches, which answered our
purpose quite well enough, we explored the cave at leisure, closely
scrutinising the walls and roof as high as we could reach, in the hope of
perhaps finding some scratch or sculptures, however rude, to prove that the
place had been inhabited in the times of the “cave-men.” Nothing of the
kind, however, was discernible. The cave in its every part is exceedingly
damp and cold, with green, slimy roof and walls, where not even the
hardiest wild beast of mountain or forest would think of taking up its abode,
far less any human being with the faintest notion of the value of warmth
and comfort. There are scores of lesser caves and fissures in the rocks
around where one would elect to live by reason of their dryness, in
preference to the big and pretentious Samuel’s Cave, which, as a mere cave,
is perhaps interesting enough, and not unworthy of a visit; otherwise it is a
“sell,” in exploring which no one can spend more than the shortest five
minutes to any good purpose. In the times of civil wars and clan feuds it is
conceivable that one or more outlawed and “broken” men might find the
outer cave a secure and not altogether unpleasant place of shelter to pass a
night in where no better might be. As a place also to hide one’s more
valuable goods and chattels in an emergency, the cave may at times have
had its value and use. It never, depend upon it, was inhabited for any length
of time by any human being. A week of it would kill the stoutest, robustest
savage that ever trod the Caledonian wilds. An additional proof, if
additional proofs are wanting, that Samuel’s Cave can never have been
“inhabited” in any proper sense of that term, or even much frequented for
any purpose whatever, is to be found in the fact that there is not a vestige of
a path either from the river bank below or from the hill above leading
towards it. Had it at any time been much in use for any purpose, there must
have been a path leading to it either from above or from below, and some
traces at least, however faint, of such a path, must still exist. We searched
and searched, above and below, and round and round, and no trace or
vestige at all of such a path could we find. Go, good reader, and see the
cave by all means when you have the opportunity; it is a fair enough cave as
caves go; but take our word for it that the attempt to invest the vast dark,
damp, slimy antrum with any archæological interest is the greatest delusion
in the world.
 CHAPTER LVIII.

Showers in Harvest Time—Magnificent Sunset—Night sometimes seeming not to

descend but to ascend—Death of M. Leverrier—The Discovery of Neptune—
Pigeon cooing at Midnight—The Owl at Noon—Cage-Birds singing at Night.

The weather continues wonderfully fine for the season [October 1877], and
with the exception of the potato-lifting, all our harvest labours are at length
concluded. The ingathering has upon the whole been highly satisfactory, far
more so than any one could have had the courage to predict up to the very
advent of this our autumnal summer, which has already lasted just thirty
days, uninterruptedly sunny and dry, without any more serious break than a
mere passing shower, which invariably did more good than harm. More
good? the reader exclaims interrogatively, how can a shower do good, how
can it be otherwise than harmful in harvest time? Patience, courteous reader,
and we shall explain. It is a case of something of this kind. You are driving
along the road; the horse in the shafts before you is upon the whole a
steady-going and willing animal enough, but you have let him have it just
his own way for the last half hour, and dreaming, perhaps, of fresh fields
and pastures green, he has for the moment forgotten your existence, and
begins to lag. His usual pace of a good eight miles an hour is now hardly
over five, and what in such a case shall you do? You drop the lash gently
across his flank, as light and gently as falls the angler’s cast on the waveless
pool; you are too much of a Christian and a gentlemen—the terms are or
ought to be synonymous—to do otherwise until it is absolutely necessary.
Your horse forgets his dream; becomes instantly alive to the work before
him; gathers himself together, and with a responsive toss of his head and a
lively play of ears, goes along at rather more than his average speed until
the next stage is reached; knowing full well that the hand that laid on that
serpent-like lash so tenderly, can lay it on in very different fashion, hot and
heavy enough when occasion calls. Or, dropping metaphor, let us state the
matter plainly, thus:—Here in Lochaber, and we suppose it is just the same
over all the Highlands, when really fine weather comes, we are for the first
few days up and doing, busy enough. But as one fine day succeeds another,
we are very ready to fall into the error that after all it is best to take things
leisurely. Where’s the need, we ask ourselves, for so much hurry and
bustle? The fine weather has lasted a week; it may last a month, is indeed
likely so to last; it is no more like rain to-day than it was yesterday; and
thus we lapse, often unconsciously, perhaps, into a spirit of dilatoriness and
procrastination, out of which only a lowering sky, and a shower that for all
we know may become a flood, can fairly rouse us. You slept long, for
instance, this morning; you dawdled over your porridge and milk at
breakfast time, and it is now noonday. But see! the heavens yonder in the
north-west are suddenly overcast; an ominous gloom creeps over the Outer
Hebrides; a few drops of rain have already fallen, one on the back of your
left hand, on which placing the index finger of your right, you can find that
it is wet, that it is rain; a second on your cheek with a soft, tepid thud; and a
third right into your open, uplifted eye, and you straightway start into
activity and life. All hands on deck! is the cry. You rush into the field
amongst the stooks; you bustle about cheerily, and calling all hands into
your service, for idlers are now out of place, you cart and carry away as fast
as you can into your barn or stack-yard, and by sunset, so expeditiously
have you worked, that the field from head-rig to head-rig is but bare and
stookless stubble. It was after all but a passing shower; the gloom has given
place to cloudless blue; you have been cheated, so to speak. But what
matters it? Your crop is safely stacked or housed, and were it not for the
passing shower and temporary mid-day gloom, your stooks were still afield,
running a risk there was no reason they should run; and so, good reader, you
will understand how a slight shower in the season of ingathering may not
always be an evil, but a very good thing indeed; and only a few such
passing, labour-inciting showers have we known here for a whole month,
and that is much to say when the month is to be counted from mid-
September to mid-October.

And, O gentle reader, we only wish you were with us here to see for
yourself, propriis oculis, for no pen can describe it, one or more of the
many magnificent sunsets we have had in the course of this same bypast
month of fine weather. The sunsets of the equinoctial seasons, both vernal
and autumnal, are almost always beautiful, more particularly those of the
autumnal equinox; but never before, we think, have we seen them so
startling, gloriously beautiful, so gorgeously magnificent, as on several
occasions lately. A few evenings ago, as we were busy in our study, a young
lady burst in upon us in a state of great excitement, begging us to throw
aside our pen for a little, and come out to see the exceeding glory of the
setting sun. We readily complied of course, and taking the young lady by
the hand we made a race of it till we reached our “coigne of vantage,” a
grassy green knoll, a favourite standpoint when any celestial phenomenon
of importance to the W. or S.W. of us is to be observed. The scene, in truth,
was indescribably beautiful, and we stood in speechless admiration, not
unmingled with awe, in sight of the most glorious sunset our eyes ever
beheld. Before us lay the whole expanse of the Linnhe Loch, shimmering as
if gently aboil in a flood of pale golden light. Beyond, rose what seemed the
one vast unbroken range of the mountains of Ardgour, Kingerloch, and
Morven, bathed in a rich dark purple hue, that for the moment so
thoroughly obliterated every trace of their native ruggedness, that our
companion prettily observed, “Haven’t you the idea, sir, as I have, that if
one were only near enough these beautiful mountains to pat them lovingly
with the hand, they would feel to the touch soft and warm as a roll of
velvet?” a thought, unconsciously perhaps, tinged with poetry, though the
woman pure and simple comes out very unmistakeably in the reference to
the “roll of velvet.” In the far background, thirty miles away, rose the glory
and pride of Mull (Blackie’s favourite island of all the Hebrides), the huge
mountains of Benmore and Ben-na-Bairnich, their base and middle zones
ink-black, their shoulders dark orange, here and there curiously streaked
with threads of pearly light, their summits and sloping ridges fringed with
living fire. Above, the whole western heavens was full of vast continents,
peninsulas, isthmuses, and islands of cloud, all afire at their edges, with
firths, ferries, and Mediterraneans of liquid gold between. As the full-orbed
sun, fiery and red, slowly sank to the horizon, the clouds were rent asunder
as if by the very excellency of the glory that beat upon them; some of them
assuming fantastic shapes, in which a lively imagination had no difficulty in
tracing striking resemblances to the hugest animals of our own and past
ages, a monster saurian in sharply defined silhouette, being so marvellously
outlined that our fair companion sketched it on the spot, as a memento of a
sunset that neither of us is likely ever to forget. As the sun’s lower limb
seemed just to touch and rest an instant on the highest peak of the
Kingerloch range, a large mass of cloud immediately above him rapidly
assumed a columnar shape, perpendicular to the plane of the horizon, and,
as the splendid orb dipped and disappeared, this huge “pillar of cloud”
became a perfect Ionic column, sharply outlined, and admirably correct in
all its proportions from base to entablature, and all aglow with living fire;
shaft and pediment with richest crimson; frieze and architrave and cornice
with the glow of molten mettle at “white heat” as it issues from a blast
furnace. There was, truth to say, something terrible about the scene, a wild
and weird combination of the sublime and beautiful such as Edmund Burke
never beheld even in his dreams. It was impossible, in the presence of the
“terrible majesty” of that glory, to avoid thinking of the awfulness that must
appertain to a scene of which all of us shall one day be spectators, when the
“elements shall melt with fervent heat,” and the “earth also, and the works
that are therein,” shall be consumed with fire. The succeeding afterglow of
that same evening was singularly beautiful. The mountains of Appin and
Glencoe were for a time bathed from their summits to their shoulders in the
richest purple and gold, making them look so soft and warm, that for the
moment their actual ruggedness was utterly forgotten, and one felt towards
them a far stronger and tenderer sentiment than mere admiration. And very
curiously, as we gazed, did the night immediately succeed the afterglow, for
of twilight there was none—there rarely is indeed in autumn, as the old
Highlanders were too observant not to notice, for what saith the old and
well-known rhyme?—

“Mar chlorich a ruith le gleann,

Tha feasgar fann, fogharaidh.”

The meaning of which is, that no longer lasts the autumnal twilight than it
takes a stone to roll adown the mountain steep into the glen below. We
generally speak of the night’s descending; we say the falling night, the
darkness fell, &c., as if the darkness came down from above, and
sometimes, doubtless, it does seem so to fall—to descend like a curtain. On
this occasion, however, and frequently, we have noticed, in the autumnal
season, the night did not seem so much to descend as to ascend, like an
exhalation from out the entrails of the earth; the blackness of gorge and
corrie and glen slowly creeping upwards, banishing the gold and purple as it
ascended, just as you have seen the earth’s shadow in an eclipse of the
moon obliterate the silvery radiance of the lunar disc—finally reaching
ridge and summit and loftiest peak, and lo, it was night, the ruddy orb of
Mars over the now ink-black top of Buachaille-Etive putting the fact
beyond all question; and, while our fair companion went for a stroll along
the beach, gaily singing a merry roundelay as became her innocence and her
years, we retired in a mood of mind that, while it was pleasant upon the
whole, had yet a tinge of sadness about it, to our study and our books.

France has recently lost one of her greatest men by the death of M.
Leverrier, her distinguished astronomer, the most distinguished astronomer,
it is not too much to say, of the present century. Many, indeed, achieved
greater triumphs with the telescope, for with the telescope Leverrier did
comparatively little; it was as a mathematical astronomer that he was
unrivalled. He came first prominently into notice while still a young man,
with his cometary investigations, and his researches into the motions of the
planet Mercury, constructing tables by which transits of the latter can be
predicted with such absolute correctness that the mean error never exceeds
sixteen seconds of time. But it is with the discovery of the planet Neptune
that Leverrier’s name is imperishably associated. The case briefly stated
was this:—It was found, after a time, that the planet Uranus, discovered by
Sir William Herschel, did not actually follow the orbit which theory had
assigned to it. It had a mysterious trick of leaving the computed track, and
describing a greater orbit, if the law of gravitation was to hold good, than
the tables founded on that law warranted. Astronomers were puzzled to
account for the vagaries of an orbit that, according to their theory, ought to
be well-behaved, and staid and steady-going as any other member of the
solar system. What could the perturbations of Uranus mean? was the
question asked; and at the suggestion of his friend the distinguished Arago,
Leverrier undertook to answer it, and in due time did answer it in such wise
as filled the world with astonishment and admiration. Resolutely grasping
with his task, Leverrier laboured long and laboured hard to resolve the
mystery, and as a first step with this result, that the problem was utterly
unresolvable on any other conceivable theory or conjecture than that
another planet, albeit unknown to astronomers, and hitherto as unsuspected
as it was unseen, existed exterior to Uranus, and that it was to the attraction
or disturbing influence of this hitherto undreamt-of orb that the
perturbations and mysterious vagaries of Uranus could alone be ascribed. A
memoir stating the conclusion arrived at, and all the calculations leading
towards it, was read before the Royal Academy of Sciences in June 1846,
and the young and daring astronomer straightway resumed his labours, of
which the aim was now to determine the elements of the orbit of the
unknown planet, in the existence of which he now believed as firmly as in
that of the visibly perturbed orb Uranus itself. The astronomical world
shook its head dubiously, and waited. Did such a planet really exist, and if it
did, could this daring Frenchman find it? M. Leverrier meantime laboured
on, and finally mastering every difficulty, he gave the computed plans of
orbit, the mass and natural position of his constructed world, if in truth, that
is, such a world existed. This was in a second memoir to the Academy of
Sciences on the last day of August 1846. Towards the end of the following
month (September 1846), Leverrier wrote to M. Galle, of Berlin, requesting
him to level the powerful telescope under his charge at a particular point of
the heavens, and there, in effect, said the wonderful Frenchman, you will
find the cause of the perturbations of Uranus, a new and distant world,
hitherto undreamt of and unseen by mortal eye, but existing all the same.
M. Galle, on the first favourable opportunity, directed his telescope as
requested, and there, within less than a single degree of its computed place,
and flinging back its light from the enormous distance of more than three
billions of miles, was the planet of Leverrier’s analysis, with a diameter,
magnitude, and orbit all as calculated and predicted. It was a glorious
triumph, the most wonderful achievement in the annals of a science where
all is wonder.

Publicly and privately has this query been put to us—Is it unusual to hear a
pigeon cooing at midnight, and the owl hooting in bright noonday? We
answer very unhesitatingly that it is unusual, so unusual in the case of the
owl at least, that in a quarter of a century’s familiar and friendly intercourse
with our wild-birds under all possible circumstances, we have never heard
an owl hoot except “darkling,” as Milton has it, that is, from out the
darkness or sombre shade. Even at night, if the moon is shining bright, it
never hoots from a spot on which the moonbeams fall in full flood; it
selects the deepest shadow even in faint moonlight when uttering its eerie
notes. It will hoot in twilight, and it will hoot when the heavens are bright
ablaze with the most brilliant coruscations of the aurora, but never, so far as
our experience has extended, does it hoot in honest daylight or even in
moonlight, except when, as we have said, it is itself in deep shade. We have
kept pets of all our native species of owls, and most interesting pets they
make, and though, when angry or in any way out of sorts, it will utter a
ready hiss, ending in a curious rasping guttural, we have never known it to
hoot except in the darkness of night, and, more rarely, in the dim, uncertain
light of evening or morning twilight. The cooing of a pigeon at midnight,
while it may be said to be unusual, is yet a thing that, under certain
circumstances, may be heard at any time. Many birds, captives in cage or
aviary, frequently sing short and incomplete strophes of their special song
in the warm stillness of summer nights, evidently in their dreams. Others, in
their natural state of freedom, about the time of the longest day, when there
is hardly any night in our latitudes, may be heard singing, generally
unconnectedly, and in a faint, uncertain key. The pigeon will coo at any
time when brooding, if rudely disturbed in any way, just as a brooding hen
will purr and scold if you annoy her or her nest at any hour of the day or
night. The cooing of a pigeon, therefore, at midnight is nothing very
wonderful. The hooting of an owl at noonday, however, is surprising, and a
thing which, although we live in a district where owls are plentiful, is
altogether unknown in our experience.
 CHAPTER LIX.

October Storms—Cablegram Predictions—Indications of coming Storms—

Geordie Braid, the St. Andrews and Newport Coach-driver—The Naturalist in
Winter—Drowned Hedgehogs: Spines become soft and gelatinous—Lophius
Piscatorius—Disproportion between head and body in the Devil-Fish a puzzle—
An Itinerant Fiddler.

The storms of the latter days of October [November 1877] were

exceedingly severe along our western seaboard, and terribly so, as more
than one correspondent assures us, amongst the Hebrides. It is worth noting
with what marvellous punctuality these Trans-atlantically telegraphed
storms reach our shores. They are “up to time,” with all the precision almost
of our best appointed mail trains; quite as punctual, at all events, to their
predicted time on several occasions lately as our ocean mail-carrying steam
ships to their appointed dates of arrival. This last October storm, for
example, was telegraphed as being due on our British shores on or about
Saturday, the 27th, and so correct, considering all the difficulties of such
meteorological vaticinations, was the prediction, that the storm actually
reached us here on the evening of the 26th, increasing in intensity
throughout the night and until mid-day of the 27th, the very day fixed upon,
when it blew with all the force of a hurricane, and the rain fell in torrents,
accompanied, too—that none of the essentials of a great storm might be
wanting—by vivid lightning, and thunder peals loud enough to make the
deafest hear, or at all events feel, for it is no exaggeration to say that the
very ground seemed at times to thrill responsive to the aërial concussion.
The 26th had dawned bright and clear, and so continued throughout the day;
one of those “pet days,” in short, not uncommon at this season,—the sea,
too, calm and glassy as a mirror. In the afternoon, however, we were called
out from the tea-table to look at a phenomenon which had already attracted
the attention of some of our more observant neighbours, and about which
they wanted our opinion, as they had some thoughts of going a herring
fishing. The phenomenon in question was this: Not a breath of air was
stirring, Loch Linnhe was unruffled by the slightest zephyr, and yet a heavy
surge quite suddenly began to break along the beach with a sudden boom
that was remarkable in such a calm. A somewhat similar phenomenon,
lasting but for a short time, however, is observed in our lochs when, on a
calm summer evening, one of the Messrs. Hutcheson’s paddle steamers—
the “Chevalier,” for instance—passes at full speed close in shore. What
could this swell and surge, troubling a loch otherwise calm as a mill-pond,
mean? You might have safely carried a lighted candle exposed and
lanternless along the beach on which that heavy swell with hollow boom
was breaking—breaking in great green waves that showed not a bell or
fleck of foam on their crests until they thundered on the shingle. It was, in a
word, a phenomenon for which there was no apparent adequate cause. The
sea, had it been in keeping with all its visible surroundings, should have
been calm and still; on the contrary, it was restless and perturbed, and there
lay the mystery. Even had we recollected nothing of the telegraphed storm,
it was easy of solution, and our instant interlocutor, as the law courts have
it, was this: “A storm in the Atlantic, my good friends. Calm as it is here,
there is a storm, and a wild one, depend upon it, outside yonder island of
Mull, for all it basks so peacefully in the golden sunset. Nothing else can
adequately account for such a swell on our calm inland waters on an
evening so summer-like and warm; and when I tell you that a storm likely
to reach our shores to-morrow has been telegraphed from America several
days since, I conclude that it is that very storm fast approaching us that
causes this swell upon our shore. It must be just at hand; so haul up your
boats high and dry; take down your nets from the drying-poles, and put
them in a place of safety. Stay thankfully at home, and let the herring
fishing stand over till the predicted gales have come and gone. Many a
gallant fellow at this moment afloat would be glad to have his foot like you
on terra firma: a chas air talamh tioram were the words,—his foot on dry
land.” With some such remarks as these, we sent the men home, still
wondering, however; and within a couple of hours the storm was upon us
with a loud prolonged shriek, that showed how thoroughly in earnest it was.
Timeously warned, no danger was done in our district, and we are now
unanimous in speaking with the utmost respect of the Atlantic cable in
connection with storm warnings from the Western Continent. These
telegraphic warnings from America, by the way, of coming storms are of
the utmost importance and value, more particularly to the western shores of
the British Islands. We have no doubt at all that on the western seaboard of
Scotland alone many valuable lives were saved, as well as much valuable
property, by the submarine cable notice that put us all on our guard with
reference to the gale that raged on the 27th of October, and for several days
subsequently. We wonder if from Britain or the Continent any of the terrible
easterly storms of last winter were telegraphed to America—timeously and
purposely telegraphed, that is—so as to be of benefit to our Transatlantic
cousins, as their recent telegrams have been to us. We fear not. But now at
least it is surely a matter of the merest courtesy and cousinly goodwill that
we be prepared and ready to send them betimes telegraphic messages of all
our easterly storms, in return for similar favours on their part in respect to
those that are westerly.

Reading over the foregoing paragraph, which the reader may see was
written currente calamo—at a gallop, as it were, and without a check, as the
foxhunter says—we find that we have used the often-quoted Latin phrase
terra firma; words which rarely fail to make us smile in their connection
with an anecdote current in St. Andrews in our early college days. It was to
this effect: The driver of a two-horse coach that ran at that time between St.
Andrews and Newport was a George Braid, a respectable old man,
familiarly known to everybody, and notably to the University students, as
“Geordie,” a liberty with his Christian name which Mr. Braid in nowise
resented, for he was intelligent and shrewd, and knew that he was thus
spoken of and addressed out of goodwill and kindly regard rather than
otherwise. Frequently patronised on his route by learned professors and
lively students, Geordie had picked up many big words and learned phrases,
which he was fond of using in his family, and, as the Catechism says,
amongst his “inferiors and equals.” In connection with frequent storm and
shipwreck on the wild east coast, it was the most natural thing in the world
that Geordie should often have heard from the lips of some of his learned
“fare” the words terra firma, with which he associated a general idea of
protection, comfort, and safety. One terrible night of snow and storm,
having driven a large coachful from Newport to the city, Geordie, when he
had duly seen to his cattle, and paid a short visit to the bar of the “Cross
Keys” hostelry, wended his way by the West Port to his home, which lay
beyond the old city walls. His wife, a brisk and eident bit body, had a
roaring fire and a cheery welcome for her goodman on his entrance, while
his children gathered round him to help him off with caps, coats, leggings,
and all the other belongings of the outer man of a driver in the good old
coaching days. Reduced at last to something like his natural dimensions,
Geordie, having sufficiently rubbed his purple hands before the fire, looked
benignantly around and exclaimed, “Ah! Meg, my woman, you and the
bairns hae muckle cause to be thankful to your Maker that ye hae terra
firma abune your heads this night! Its just awfu’ out yonder by the Guard
Brig and Strathtyrum.” We have met with not a few in our day with a
strange craze for using words and phrases of which they evidently knew as
little of the real meaning and proper application as honest Geordie Braid
with his terra firma.

The new moon of the 5th, aided by a wind that at times almost amounted to
a gale, gave us along the western seaboard three very high tides in
succession; that of the afternoon of the 6th, however, being the highest. The
naturalist who is fairly diligent on such occasions is pretty sure to meet with
more or less interesting matter for thoughtful study; nor, so far as our own
experience extends, need the entries in one’s note-book, even for what is
called the “dead” season of mid-winter, be fewer in number, or less
interesting or instructive than those of the pages devoted to the summer
season itself. We have known naturalists whose note-books presented little
but a dreary succession of blank pages for the winter half-year, and who
thought it odd that we should be surprised at it. It has been said that the
laws of disease are as beautiful as those of health, and that peace has its
victories as well as war, and we have no hesitation in saying that to the true
naturalist the winter season, if fairly and diligently encountered, is in its
way just as interesting as the summer, and that the observer who has all his
wits about him, and who goes to work with a will, may have his “victories”
even in the season of the winter solstice—victories as important in their
way and gratifying as are those of midsummer itself, when the days are at
their longest, when summer seas are calm and summer woods are green. In
the course of half an hour’s ramble on the beach the other day, we fell in
with some curious waifs, each of which might be made the text of an
interesting monograph. Three drowned hedgehogs, for example, was a
somewhat startling “find” to turn up in a swathe of seaware that the
advancing tide was slowly rolling up the shingle. One was full-grown, a
female; the other two, both males, were but half or three parts grown. What
brought them there? was the natural question; for a hedgehog, dead or
living, on the sea-shore under high-water mark, is as odd and out-of-place
an object as would be a mackerel far up the hills amongst the heather. The
following is probably a satisfactory enough explanation of the mystery:—
Hedgehogs, which twenty years ago were quite unknown in Lochaber, are
now plentiful. A pair, captured on Lord Abinger’s lands at Torlundy, were
sent to us some dozen or fifteen years ago as a great curiosity; and in this
district then they were a curiosity, so much so, that we can recollect that
during the time they remained in our possession as exceedingly tame and
most interesting pets, people from all parts of the country used to come in
order to have a close look at the black-snouted, spine-armoured hedge pigs,
as Shakespeare calls them, the graineag or repulsive one of the midland
Highlanders of Athole and Strathspey, where the animal has always been
plentiful. They have now become so common in this district that a
hedgehog is no more accounted a rarity than is a stoat or a weasel.
Hedgehogs are fond of making their cozy nests of moss, grass fibres, and
fallen leaves, near the roots of trees and bushes growing on the banks of
rivers and mountain streams. These last have of late been frequently
swollen beyond their usual bounds by the heavy rains; and in a spate of this
kind poor Mrs. Hedgehog and her youngsters were caught napping, and
carried away by the torrent to the sea, and ultimately cast ashore by the
wind and waves, where we found them in their winding-sheet of slimy sea-
wrack, and for a moment wondered how it came to pass that they lay there,
like poor Ophelia, “drown’d, drown’d.” One remarkable circumstance
connected with these drowned hedgehogs was this: we found to our surprise
that we could handle them with impunity; their spines, so formidable in the
living animal, being quite soft and gelatinous to their very tips. This is by
no means the case with the spines of such hedgehogs as are killed by trap,
or otherwise on land. In this latter case the spines retain their point and
prickliness, as in the living animal, till in the process of decay they separate
from their sockets in the skin, and drop in brittle, broken fragments to the
ground. A question, then, for future investigation is this,—Do the spines of
all drowned hedgehogs lose their prickliness and point, and become soft
and gelatinous? If so, has fresh water alone this effect, or is it necessary that
the animal should be some time immersed in salt water?

Within a short distance of the drowned hedgehogs, lay a large angler or

fishing-frog, the Lophius piscatorius of ichthyologists, and a frequent waif
on our shores after a gale. It had evidently been caught by the storm in
shallow water, and been beaten to death by the weight and force of the
waves, for it was in excellent condition, and there was nothing to indicate
death in any other way. Why in this fish such a huge head, with its
formidable array of recurved teeth, and such a cavernous, capacious gullet,
should be joined to a body comparatively so diminutive, is a puzzle that has
never yet been satisfactorily solved; nor can we ourselves, up to this present
moment, advance even a plausible conjecture in explanation of an anomaly
that must have attracted the attention of thousands. The disproportion
between the immense head and the small and slender body is as great as if
you erected a porch lofty and wide enough to serve as the main entrance to
a cathedral, and vestibule to correspond, in order to enter a dwelling
consisting after all but of a single bedroom. Or, to put it in another way, it is
as if you built a large mill, with the most powerful machinery possible, in
order to grind sufficient meal for the daily consumption of a single
dyspeptic customer. The fishing-frog, has, we believe, been of late
successfully introduced into more than one of our many aquaria, but we are
not aware that any satisfactory explanation of the difficulty which we are
considering has as yet been arrived at. A full and sufficient explanation,
however, you may be sure there must be, if we only know enough of the
animal’s economy to get at it.

But we must stop; for hark! an itinerant fiddler has this moment struck up
“Bob of Fettercairn” just in front of our study window. He plays admirably
too, lovingly caressing the polished base of his instrument—his bread-
winner, poor fellow—with his wan and withered cheek, and wielding a
powerful, yet light and delicate, bow-hand; and we must go and have a
crack with him. Nor must you sneer at us for so doing, gracious reader. The
arrival even of a peripatetic, out-at-elbows fiddler is an event of some
importance in such a place as this on a cold, grey November afternoon. We
shall order him a big bowl of tea, with something to eat, satisfied that if in
so doing we are not entertaining an angel unawares (though there is no
reason that we know of why an angel should not appear in peripatetic
fiddler guise, as well as in any other form), we are at all events entertaining
one who by his appearance manifestly needs something warm and
comfortable, and a little rest by a cheerful fireside at this season, not
forgetting the while that he is a capital fiddler—of some intelligence, too,
and full of capital stories we warrant him. Depend upon it that Homer, who
was after all but an inspired gaberlunzie, has many a time and oft appeared
in quite as sorry a plight, and with as little externally to recommend him as
this same itinerant fiddler; and think how proud and glad you and we should
be to have a chance of entertaining the blind old Chian, wandering ballad-
singer as he was! You must, therefore, let us have our way with this poor
old man, who, by the way, in not blind, but, on the contrary, has a good
large dark brown eye of his own, so common, we have noticed, in people
musically inclined, that it may be called the musical eye; and if he is all we
take him for, and he and we got on well together, you may perhaps hear of
him again.
 CHAPTER LX.

A Trip to Glasgow—Kelvin Grove Museum—Highland Association—A run to

Rothesay—Rothesay Aquarium.

Favoured by the most splendid Christmas weather [January 1878],

piercingly cold, indeed, but beautifully bright and clear, a run from
Lochaber to Clydesdale on an agreeable errand is exceedingly enjoyable.
Our first day in Glasgow was devoted to the Kelvin Grove Museum, which
we had now an opportunity, for the first time, of examining thoroughly and
at leisure, and with which, as the reader may believe, we were very much
delighted. On handing our card to Mr. Paton, the curator, we were received
by himself and his assistant, Mr. Campbell—the latter, of course, a
Highlander—in the friendliest manner; and a couple of hours were very
pleasantly and profitably spent in examining a really curious and valuable
collection, so admirably catalogued and arranged, that we believe we saw
and minutely studied everything to be seen as leisurely and satisfactorily as
was possible in the time at our disposal. Our friend Mr. Snowie, of
Inverness, had written us before leaving home that he was sending some
contributions to the museum, of which he begged us to undertake the formal
delivery, and see properly placed; and this of course we had much pleasure
in doing. These contributions are a valuable acquisition to the museum, and
are as follows:—(1.) Hoopoe (Upupa epops, Linn.), a female, in fine
plumage, and admirably set up. This bird was captured by the boys at the
Inverness Reformatory School, and dying, notwithstanding it received all
the attention and kindly care that could be bestowed upon it, it passed into
Mr. Snowie’s hands. (2.) Wild cat, stuffed, an excellent specimen, with very
prominent markings, trapped at Fasnakyle, on The Chisholm’s estate. (3.) A
white blackbird, and an albino bunting, both shot by Mr. T. B. Snowie near
Inverness. (4.) Snipe and other marsh-bird skins, shot by the same. (5.) Two
small hares preserved in a bottle; taken out of an unusually large-sized
female shot at Dochfour in September 1875; a very interesting preparation.
(6.) Head of otter, trapped on the River Peffer in 1876. (7.) Owl (Strix
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com