21 October 2020

CLUSTERXL

R81

Administration Guide
[Classification: Protected]
Check Point Copyright Notice
© 2020 Check Point Software Technologies Ltd.

All rights reserved. This product and related documentation are protected by copyright and distributed
under licensing restricting their use, copying, distribution, and decompilation. No part of this product or
related documentation may be reproduced in any form or by any means without prior written authorization
of Check Point. While every precaution has been taken in the preparation of this book, Check Point
assumes no responsibility for errors or omissions. This publication and features described herein are
subject to change without notice.

RESTRICTED RIGHTS LEGEND:

Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)
(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013 and FAR
52.227-19.

TRADEMARKS:
Refer to the Copyright page for a list of our trademarks.
Refer to the Third Party copyright notices for a list of relevant copyrights and third-party licenses.
 ClusterXL R81 Administration Guide

Important Information
Latest Software
We recommend that you install the most recent software release to stay up-to-date with the
latest functional improvements, stability fixes, security enhancements and protection
against new and evolving attacks.

Certifications
For third party independent certification of Check Point products, see the Check Point
Certifications page.

Check Point R81

For more about this release, see the R81 home page.

Latest Version of this Document in English

Open the latest version of this document in a Web browser.
Download the latest version of this document in PDF format.

Feedback
Check Point is engaged in a continuous effort to improve its documentation.
Please help us by sending your comments.

ClusterXL R81 Administration Guide      |      3

 ClusterXL R81 Administration Guide

Revision History

Date Description

21 October 2020 First release of this document

ClusterXL R81 Administration Guide      |      4

 Table of Contents

Table of Contents
Glossary 12
Introduction to ClusterXL 35
The Need for Clusters 35
ClusterXL Solution 35
How ClusterXL Works 35
The Cluster Control Protocol 36
ClusterXL Requirements and Compatibility 37
Check Point Appliances and Open Servers 37
Supported Number of Cluster Members 38
Hardware Requirements for Cluster Members 38
Software Requirements for Cluster Members 38
VMAC Mode 39
Supported Topologies for Synchronization Network 41
Clock Synchronization in ClusterXL 44
IPv6 Support for ClusterXL 44
Synchronized Cluster Restrictions 44
High Availability and Load Sharing Modes in ClusterXL 46
Introduction to High Availability and Load Sharing modes 46
High Availability 46
Load Sharing 47
Example ClusterXL Topology 48
Example Diagram 48
Defining the Cluster Member IP Addresses 49
Defining the Cluster Virtual IP Addresses 49
Defining the Synchronization Network 49
Configuring Cluster Addresses on Different Subnets 50
ClusterXL Mode Considerations 51
Choosing High Availability, Load Sharing, or Active-Active mode 51
Considerations for the Load Sharing Mode 51
IP Address Migration 51

ClusterXL R81 Administration Guide      |      5

 Table of Contents

ClusterXL Modes 52
High Availability Mode 52
Load Sharing Modes 55
Load Sharing Multicast Mode 55
Load Sharing Unicast Mode 56
ClusterXL Mode Comparison 58
Cluster Failover 59
What is Failover? 59
When Does a Failover Occur? 59
What Happens When a Cluster Member Recovers? 60
How a Recovered Cluster Member Obtains the Security Policy 60
General Failover Limitations 61
Active-Active Mode in ClusterXL 62
Introduction 62
Configuring Active-Active mode 63
Dynamic Routing Failover 66
Limitations 69
Synchronizing Connections in the Cluster 70
The Synchronization Network 71
How State Synchronization Works 72
Configuring Services to Synchronize After a Delay 73
Configuring Services not to Synchronize 74
Sticky Connections 76
Introduction to Sticky Connections 76
VPN Tunnels with 3rd Party Peers and Load Sharing 76
Third-Party Gateways in Hub and Spoke VPN Deployments 77
Configuring a Third-Party Gateway in a Hub and Spoke VPN Deployment 78
Non-Sticky Connections 80
Non-Sticky Connection Example: TCP 3-Way Handshake 80
Synchronizing Non-Sticky Connections 81
Synchronizing Clusters on a Wide Area Network 82
Synchronized Cluster Restrictions 84
Configuring ClusterXL 85

ClusterXL R81 Administration Guide      |      6

 Table of Contents

Installing Cluster Members 85

Configuring Routing for Client Computers 85
Configuring the Cluster Control Protocol (CCP) Settings 86
Configuring the Cluster Object and Members 87
Using the Wizard Mode in SmartConsole 87
Using the Classic Mode in SmartConsole 91
Changing the Settings of Cluster Object in SmartConsole 95
Configuring General Properties 95
Working with Cluster Topology 95
Changing the Synchronization Interface 97
Adding Another Member to an Existing Cluster 98
Removing a Member from an Existing Cluster 98
Configuring a ClusterXL in Bridge Mode 99
Advanced Features and Procedures 100
Working with VPN in Cluster 101
Configuring VPN in Clusters 101
Defining VPN Peer Clusters with Separate Management Servers 102
Working with NAT in Cluster 103
Cluster Fold and Cluster Hide 103
Configuring NAT in Cluster 103
Configuring NAT on a Cluster Member 103
Working with VLANs in Cluster 105
Configuring Link Monitoring on the Cluster Interfaces 107
Working with Bond Interfaces in Cluster 110
Bond Interfaces (Link Aggregation) 111
Bond High Availability Mode in Cluster 113
Simple Redundant Topology 114
Fully Meshed Redundancy 115
Bond Failover in High Availability Mode 116
Configuring a Bond Interface in High Availability Mode 117
Making Sure the Bond Interface is Functioning Properly 118
Failover Support for VLANs 119
Sync Redundancy 120

ClusterXL R81 Administration Guide      |      7

 Table of Contents

Configuring Bond High Availability in VRRP Cluster 121

Bond Load Sharing Mode in Cluster 123
Bond Failover in Load Sharing Mode 124
Configuring a Bond Interface in High Availability Mode 125
Configuring Minimal Number of Required Slave Interfaces 126
Making Sure the Bond Interface is Functioning Properly 127
Group of Bonds 128
Introduction 128
Creating a new Group of Bonds 130
Adding a Bond interface to the existing Group of Bonds 131
Removing a Bond interface from the existing Group of Bonds 133
Deleting a Group of Bonds 134
Monitoring 135
Logs 135
Limitations 136
Performance Guidelines for Bond Interfaces 137
Troubleshooting Issues with Bonded Interfaces 139
Troubleshooting Workflow 139
Connectivity Delays on Switches 139
Advanced Cluster Configuration 140
Controlling the Clustering and Synchronization Timers 141
Blocking New Connections Under Load 142
Defining Non-Monitored Interfaces 144
Configuring Policy Update Timeout 145
Enhanced 3-Way TCP Handshake Enforcement 146
Cluster IP Addresses on Different Subnets 148
Configuring Cluster Addresses on Different Subnets 149
Example of Cluster IP Addresses on Different Subnets 150
Limitations of Cluster Addresses on Different Subnets 152
Connectivity Between Cluster Members 152
Load Sharing Multicast Mode with "Semi-Supporting" Hardware 152
Manual Proxy ARP 152
Connecting to the Cluster Members from the Cluster Network 152

ClusterXL R81 Administration Guide      |      8

 Table of Contents

Configuring Anti-Spoofing 153

Adding Another Member to an Existing Cluster 154
Adding a New Cluster Member to the Cluster Object 154
Adding an Existing Security Gateway as a Cluster Member to the Cluster Object 157
Removing a Member from an Existing Cluster 160
ISP Redundancy on a Cluster 163
Introduction 163
ISP Redundancy Modes 165
Outgoing Connections 165
Incoming Connections 166
Configuring ISP Redundancy on a Cluster 167
ISP Redundancy and VPN 172
Controlling ISP Redundancy from CLI 173
Force ISP Link State 173
The ISP Redundancy Script 173
Dynamic Routing Protocols in a Cluster Deployment 174
Router IP Address 174
Routing Table Synchronization 174
Wait for Clustering 174
Failure Recovery 175
ClusterXL Configuration Commands 176
Configuring the Cluster Member ID Mode in Local Logs 180
Registering a Critical Device 181
Unregistering a Critical Device 183
Reporting the State of a Critical Device 184
Registering Critical Devices Listed in a File 185
Unregistering All Critical Devices 187
Configuring the Cluster Control Protocol (CCP) Settings 188
Initiating Manual Cluster Failover 189
Configuring the Minimal Number of Required Slave Interfaces for Bond Load Sharing 193
Configuring the Multi-Version Cluster Mechanism 196
Monitoring and Troubleshooting Clusters 197
ClusterXL Monitoring Commands 198

ClusterXL R81 Administration Guide      |      9

 Table of Contents

Viewing Cluster State 203

Viewing Critical Devices 207
Viewing Cluster Interfaces 214
Viewing Bond Interfaces 218
Viewing Cluster Failover Statistics 223
Viewing Software Versions on Cluster Members 225
Viewing Delta Synchronization 226
Viewing IGMP Status 232
Viewing Cluster Delta Sync Statistics for Connections Table 233
Viewing Cluster IP Addresses 234
Viewing the Cluster Member ID Mode in Local Logs 235
Viewing Interfaces Monitored by RouteD 236
Viewing Roles of RouteD Daemon on Cluster Members 237
Viewing Cluster Correction Statistics 238
Viewing the Cluster Control Protocol (CCP) Settings 240
Viewing Latency and Drop Rate of Interfaces 241
Viewing the State of the Multi-Version Cluster Mechanism 242
Viewing Full Connectivity Upgrade Statistics 243
Monitoring Cluster Status in SmartConsole 244
Background 244
ClusterXL Log Messages 244
General Logs 245
State Logs 245
Critical Device Logs 246
Interface Logs 246
Reason Strings 247
Working with SNMP Traps 249
How to Initiate Cluster Failover 250
Troubleshooting Issues with the Critical Device "routed" 251
Background 251
Standard Behavior of the Critical Device "routed" 251
Basic Troubleshooting Steps 252
ClusterXL Error Messages 253

ClusterXL R81 Administration Guide      |      10

 Table of Contents

Command Line Reference 254

ClusterXL Configuration Commands 255
ClusterXL Monitoring Commands 258
cpconfig 263
cphastart 266
cphastop 267
cp_conf fullha 268
cp_conf ha 269
fw hastat 270
fwboot ha_conf 271
ClusterXL Scripts 272
The clusterXL_admin Script 273
The clusterXL_monitor_ips Script 277
The clusterXL_monitor_process Script 281
Cluster Management APIs 285
Introduction 285
List of APIs 285
API Examples 285
Known Limitations 299

ClusterXL R81 Administration Guide      |      11

 Glossary

Glossary
3

3rd party Cluster

Cluster of Check Point Security Gateways that work together in a redundant
configuration. These Check Point Security Gateways are installed on X-Series XOS, or
IPSO OS. VRRP Cluster on Gaia OS is also considered a 3rd party cluster. The 3rd
party cluster handles the traffic, and Check Point Security Gateways perform only State
Synchronization.

Active
State of a Cluster Member that is fully operational: (1) In ClusterXL, this applies to the
state of the Security Gateway component (2) In 3rd party / OPSEC cluster, this applies
to the state of the cluster State Synchronization mechanism.

Active-Active
A cluster mode (in R80.40 and higher versions), where cluster members are located in
different geographical areas (different sites, different cloud availability zones). This
mode supports the configuration of IP addresses from different subnets on all cluster
interfaces, including the Sync interfaces. Each cluster member inspects all traffic routed
to it and synchronizes the recorded connections to its peer cluster members. The traffic
is not balanced between the cluster members.

Active Up
ClusterXL in High Availability mode that was configured as Maintain current active
Cluster Member in the cluster object in SmartConsole: (1) If the current Active member
fails for some reason, or is rebooted (for example, Member_A), then failover occurs
between Cluster Members - another Standby member will be promoted to be Active (for
example, Member_B). (2) When former Active member (Member_A) recovers from a
failure, or boots, the former Standby member (Member_B) will remain to be in Active
state (and Member_A will assume the Standby state).

ClusterXL R81 Administration Guide      |      12

 Glossary

Active(!)
In ClusterXL, state of the Active Cluster Member that suffers from a failure. A problem
was detected, but the Cluster Member still forwards packets, because it is the only
member in the cluster, or because there are no other Active members in the cluster. In
any other situation, the state of the member is Down. Possible states: ACTIVE(!),
ACTIVE(!F) - Cluster Member is in the freeze state, ACTIVE(!P) - This is the Pivot
Cluster Member in Load Sharing Unicast mode, ACTIVE(!FP) - This is the Pivot Cluster
Member in Load Sharing Unicast mode and it is in the freeze state.

Active/Active
See "Load Sharing".

Active/Standby
See "High Availability".

Administrator
A user with permissions to manage Check Point security products and the network
environment.

API
In computer programming, an application programming interface (API) is a set of
subroutine definitions, protocols, and tools for building application software. In general
terms, it is a set of clearly defined methods of communication between various software
components.

Appliance
A physical computer manufactured and distributed by Check Point.

ARP Forwarding
Forwarding of ARP Request and ARP Reply packets between Cluster Members by
encapsulating them in Cluster Control Protocol (CCP) packets. Introduced in R80.10
version. For details, see sk111956.

ClusterXL R81 Administration Guide      |      13

 Glossary

Backup
(1) In VRRP Cluster on Gaia OS - State of a Cluster Member that is ready to be
promoted to Master state (if Master member fails). (2) In VSX Cluster configured in
Virtual System Load Sharing mode with three or more Cluster Members - State of a
Virtual System on a third (and so on) VSX Cluster Member. (3) A Cluster Member or
Virtual System in this state does not process any traffic passing through cluster.

Blocking Mode
Cluster operation mode, in which Cluster Member does not forward any traffic (for
example, caused by a failure).

Bond
A virtual interface that contains (enslaves) two or more physical interfaces for
redundancy and load sharing. The physical interfaces share one IP address and one
MAC address. See "Link Aggregation".

Bonding
See "Link Aggregation".

Bridge Mode
A Security Gateway or Virtual System that works as a Layer 2 bridge device for easy
deployment in an existing topology.

CA
Certificate Authority. Issues certificates to gateways, users, or computers, to identify
itself to connecting entities with Distinguished Name, public key, and sometimes IP
address. After certificate validation, entities can send encrypted data using the public
keys in the certificates.

CCP
See "Cluster Control Protocol".

ClusterXL R81 Administration Guide      |      14

 Glossary

Certificate
An electronic document that uses a digital signature to bind a cryptographic public key
to a specific identity. The identity can be an individual, organization, or software entity.
The certificate is used to authenticate one identity to another.

CGNAT
Carrier Grade NAT. Extending the traditional Hide NAT solution, CGNAT uses
improved port allocation techniques and a more efficient method for logging. A CGNAT
rule defines a range of original source IP addresses and a range of translated IP
addresses. Each IP address in the original range is automatically allocated a range of
translated source ports, based on the number of original IP addresses and the size of
the translated range. CGNAT port allocation is Stateless and is performed during policy
installation. See sk120296.

Cluster
Two or more Security Gateways that work together in a redundant configuration - High
Availability, or Load Sharing.

Cluster Control Protocol

Proprietary Check Point protocol that runs between Cluster Members on UDP port
8116, and has the following roles: (1) State Synchronization (Delta Sync), (2) Health
checks (state of Cluster Members and of cluster interfaces): Health-status Reports,
Cluster-member Probing, State-change Commands, Querying for cluster membership.
Note: CCP is located between the Check Point Firewall kernel and the network
interface (therefore, only TCPdump should be used for capturing this traffic). Acronym:
CCP.

Cluster Correction Layer

Proprietary Check Point mechanism that deals with asymmetric connections in Check
Point cluster. The CCL provides connections stickiness by "correcting" the packets to
the correct Cluster Member: In most cases, the CCL makes the correction from the
CoreXL SND; in some cases (like Dynamic Routing, or VPN), the CCL makes the
correction from the Firewall or SecureXL. Acronym: CCL.

Cluster Interface
An interface on a Cluster Member, whose Network Type was set as Cluster in
SmartConsole in cluster object. This interface is monitored by cluster, and failure on this
interface will cause cluster failover.

Cluster Member
A Security Gateway that is part of a cluster.

ClusterXL R81 Administration Guide      |      15

 Glossary

Cluster Mode
Configuration of Cluster Members to work in these redundant modes: (1) One Cluster
Member processes all the traffic - High Availability or VRRP mode (2) All traffic is
processed in parallel by all Cluster Members - Load Sharing.

Cluster Topology
Set of interfaces on all members of a cluster and their settings (Network Objective, IP
address/Net Mask, Topology, Anti-Spoofing, and so on).

ClusterXL
Cluster of Check Point Security Gateways that work together in a redundant
configuration. The ClusterXL both handles the traffic and performs State
Synchronization. These Check Point Security Gateways are installed on Gaia OS: (1)
ClusterXL supports up to 5 Cluster Members, (2) VRRP Cluster supports up to 2 Cluster
Members, (3) VSX VSLS cluster supports up to 13 Cluster Members. Note: In ClusterXL
Load Sharing mode, configuring more than 4 Cluster Members significantly decreases
the cluster performance due to amount of Delta Sync traffic.

CoreXL
A performance-enhancing technology for Security Gateways on multi-core processing
platforms. Multiple Check Point Firewall instances are running in parallel on multiple
CPU cores.

CoreXL Firewall Instance

Also CoreXL FW Instance. On a Security Gateway with CoreXL enabled, the Firewall
kernel is copied multiple times. Each replicated copy, or firewall instance, runs on one
processing CPU core. These firewall instances handle traffic at the same time, and
each firewall instance is a complete and independent firewall inspection kernel.

CoreXL SND
Secure Network Distributer. Part of CoreXL that is responsible for: Processing incoming
traffic from the network interfaces; Securely accelerating authorized packets (if
SecureXL is enabled); Distributing non-accelerated packets between Firewall kernel
instances (SND maintains global dispatching table, which maps connections that were
assigned to CoreXL Firewall instances). Traffic distribution between CoreXL Firewall
instances is statically based on Source IP addresses, Destination IP addresses, and the
IP 'Protocol' type. The CoreXL SND does not really "touch" packets. The decision to
stick to a particular FWK daemon is done at the first packet of connection on a very high
level, before anything else. Depending on the SecureXL settings, and in most of the
cases, the SecureXL can be offloading decryption calculations. However, in some other
cases, such as with Route-Based VPN, it is done by FWK daemon.

ClusterXL R81 Administration Guide      |      16

 Glossary

CPHA
General term in Check Point Cluster that stands for Check Point High Availability
(historic fact: the first release of ClusterXL supported only High Availability) that is used
only for internal references (for example, inside kernel debug) to designate ClusterXL
infrastructure.

CPUSE
Check Point Upgrade Service Engine for Gaia Operating System. With CPUSE, you
can automatically update Check Point products for the Gaia OS, and the Gaia OS itself.
For details, see sk92449.

Critical Device
Also known as a Problem Notification, or pnote. A special software device on each
Cluster Member, through which the critical aspects for cluster operation are monitored.
When the critical monitored component on a Cluster Member fails to report its state on
time, or when its state is reported as problematic, the state of that member is
immediately changed to Down. The complete list of the configured critical devices
(pnotes) is printed by the 'cphaprob -ia list' command or 'show cluster members pnotes
all' command.

DAIP Gateway
A Dynamically Assigned IP (DAIP) Security Gateway is a Security Gateway where the
IP address of the external interface is assigned dynamically by the ISP.

Data Type
A classification of data. The Firewall classifies incoming and outgoing traffic according
to Data Types, and enforces the Policy accordingly.

Database
The Check Point database includes all objects, including network objects, users,
services, servers, and protection profiles.

Dead
State reported by a Cluster Member when it goes out of the cluster (due to 'cphastop'
command (which is a part of 'cpstop'), or reboot).

ClusterXL R81 Administration Guide      |      17

 Glossary

Decision Function
A special cluster algorithm applied by each Cluster Member on the incoming traffic in
order to decide, which Cluster Member should process the received packet. Each
Cluster Members maintains a table of hash values generated based on connections
tuple (source and destination IP addresses/Ports, and Protocol number).

Delta Sync
Synchronization of kernel tables between all working Cluster Members - exchange of
CCP packets that carry pieces of information about different connections and operations
that should be performed on these connections in relevant kernel tables. This Delta
Sync process is performed directly by Check Point kernel. While performing Full Sync,
the Delta Sync updates are not processed and saved in kernel memory. After Full Sync
is complete, the Delta Sync packets stored during the Full Sync phase are applied by
order of arrival.

Delta Sync Retransmission

It is possible that Delta Sync packets will be lost or corrupted during the Delta Sync
operations. In such cases, it is required to make sure the Delta Sync packet is re-sent.
The Cluster Member requests the sending Cluster Member to retransmit the
lost/corrupted Delta Sync packet. Each Delta Sync packet has a sequence number. The
sending member has a queue of sent Delta Sync packets. Each Cluster Member has a
queue of packets sent from each of the peer Cluster Members. If, for any reason, a Delta
Sync packet was not received by a Cluster Member, it can ask for a retransmission of
this packet from the sending member. The Delta Sync retransmission mechanism is
somewhat similar to a TCP Window and TCP retransmission mechanism. When a
member requests retransmission of Delta Sync packet, which no longer exists on the
sending member, the member prints a console messages that the sync is not complete.

Distributed Deployment
The Check Point Security Gateway and Security Management Server products are
deployed on different computers.

Domain
A network or a collection of networks related to an entity, such as a company, business
unit or geographical location.

Domain Log Server

A Log Server for a specified Domain, as part of a Multi-Domain Log Server. It stores and
processes logs from Security Gateways that are managed by the corresponding Domain
Management Server. Acronym: DLS.

ClusterXL R81 Administration Guide      |      18

 Glossary

Down
State of a Cluster Member during a failure when one of the Critical Devices reports its
state as "problem": In ClusterXL, applies to the state of the Security Gateway
component; in 3rd party / OPSEC cluster, applies to the state of the State
Synchronization mechanism. A Cluster Member in this state does not process any traffic
passing through cluster.

Dying
State of a Cluster Member as assumed by peer members, if it did not report its state for
0.7 second.

Expert Mode
The name of the full command line shell that gives full system root permissions in the
Check Point Gaia operating system.

External Network
Computers and networks that are outside of the protected network.

External Users
Users defined on external servers. External users are not defined in the Security
Management Server database or on an LDAP server. External user profiles tell the
system how to identify and authenticate externally defined users.

Failback in Cluster
Also, Fallback. Recovery of a Cluster Member that suffered from a failure. The state of a
recovered Cluster Member is changed from Down to either Active, or Standby
(depending on Cluster Mode).

Failed Member
A Cluster Member that cannot send or accept traffic because of a hardware or software
problem.

ClusterXL R81 Administration Guide      |      19

 Glossary

Failover
Also, Fail-over. Transferring of a control over traffic (packet filtering) from a Cluster
Member that suffered a failure to another Cluster Member (based on internal cluster
algorithms).

Failure
A hardware or software problem that causes a Security Gateway to be unable to serve
as a Cluster Member (for example, one of cluster interface has failed, or one of the
monitored daemon has crashed). Cluster Member that suffered from a failure is declared
as failed, and its state is changed to Down (a physical interface is considered Down
only if all configured VLANs on that physical interface are Down).

Firewall
The software and hardware that protects a computer network by analyzing the incoming
and outgoing network traffic (packets).

Flapping
Consequent changes in the state of either cluster interfaces (cluster interface flapping),
or Cluster Members (Cluster Member flapping). Such consequent changes in the state
are seen in the 'Logs & Monitor' > 'Logs' (if in SmartConsole > cluster object, the cluster
administrator set the 'Track changes in the status of cluster members' to 'Log').

ClusterXL R81 Administration Guide      |      20

 Glossary

Flush and ACK

Also, FnA, F&A. Cluster Member forces the Delta Sync packet about the incoming
packet and waiting for acknowledgments from all other Active members and only then
allows the incoming packet to pass through. In some scenarios, it is required that some
information, written into the kernel tables, will be Sync-ed promptly, or else a race
condition can occur. The race condition may occur if a packet that caused a certain
change in kernel tables left Member_A toward its destination and then the return packet
tries to go through Member_B. In general, this kind of situation is called asymmetric
routing. What may happen in this scenario is that the return packet arrives at Member_B
before the changes induced by this packet were Sync-ed to this Member_B. Example of
such a case is when a SYN packet goes through Member_A, causing multiple changes
in the kernel tables and then leaves to a server. The SYN-ACK packet from a server
arrives at Member_B, but the connection itself was not Sync-ed yet. In this condition, the
Member_B will drop the packet as an Out-of-State packet (First packet isn't SYN). In
order to prevent such conditions, it is possible to‎ use ‎the‎"Flush ‎and ‎ACK" (F&A)
mechanism. This mechanism can send the Delta Sync packets with all the changes
accumulated so far in the Sync buffer to the other Cluster Members, hold the original
packet that induced these changes and wait for acknowledgment from all other (Active)
Cluster Members that they received the information in the Delta Sync packet. When all
acknowledgments arrived, the mechanism will release the held original packet. This
ensures that by the time the return packet arrived from a server at the cluster, all the
Cluster Members are aware of the connection. F&A is being operated at the end of the
Inbound chain and at the end of the Outbound chain (it is more common at the
Outbound).

Forwarding
Process of transferring of an incoming traffic from one Cluster Member to another
Cluster Member for processing. There are two types of forwarding the incoming traffic
between Cluster Members - Packet forwarding and Chain forwarding. Also see
"Forwarding Layer in Cluster" and "ARP Forwarding in Cluster".

ClusterXL R81 Administration Guide      |      21

 Glossary

Forwarding Layer
The Forwarding Layer is a ClusterXL mechanism that allows a Cluster Member to pass
packets to peer Cluster Members, after they have been locally inspected by the firewall.
This feature allows connections to be opened from a Cluster Member to an external
host. Packets originated by Cluster Members are hidden behind the Cluster Virtual IP
address. Thus, a reply from an external host is sent to the cluster, and not directly to the
source Cluster Member. This can pose problems in the following situations: (1) The
cluster is working in High Availability mode, and the connection is opened from the
Standby Cluster Member. All packets from the external host are handled by the Active
Cluster Member, instead. (2) The cluster is working in a Load Sharing mode, and the
decision function has selected another Cluster Member to handle this connection. This
can happen since packets directed at a Cluster IP address are distributed between
Cluster Members as with any other connection. If a Cluster Member decides, upon the
completion of the firewall inspection process, that a packet is intended for another
Cluster Member, it can use the Forwarding Layer to hand the packet over to that Cluster
Member. In High Availability mode, packets are forwarded over a Synchronization
network directly to peer Cluster Members. It is important to use secured networks only,
as encrypted packets are decrypted during the inspection process, and are forwarded
as clear-text (unencrypted) data. In Load Sharing mode, packets are forwarded over a
regular traffic network. Packets that are sent on the Forwarding Layer use a special
source MAC address to inform the receiving Cluster Member that they have already
been inspected by another Cluster Member. Thus, the receiving Cluster Member can
safely hand over these packets to the local Operating System, without further inspection.

Full High Availability

Also, Full HA Cluster Mode. A special Cluster Mode (supported only on Check Point
appliances running Gaia OS or SecurePlatform OS, where each Cluster Member also
runs as a Security Management Server. This provides redundancy both between
Security Gateways (only High Availability is supported) and between Security
Management Servers (only High Availability is supported - see sk39345).

ClusterXL R81 Administration Guide      |      22

 Glossary

Full Sync
Process of full synchronization of applicable kernel tables by a Cluster Member from the
working Cluster Member(s) when it tries to join the existing cluster. This process is
meant to fetch ‎a‎"snapshot" ‎of the applicable kernel tables of already Active Cluster
Member(s). Full Sync is performed during the initialization of Check Point software
(during boot process, the first time the Cluster Member runs policy installation, during
'cpstart', during 'cphastart'). Until the Full Sync process completes successfully, this
Cluster Member remains in the Down state, because until it is fully synchronized with
other Cluster Members, it cannot function as a Cluster Member. Meanwhile, the Delta
Sync packets continue to arrive, and the Cluster Member that tries to join the existing
cluster, stores them in the kernel memory until the Full Sync completes. The whole Full
Sync process is performed by fwd daemons on TCP port 256 over the Sync network (if it
fails over the Sync network, it tries the other cluster interfaces). The information is sent
by fwd daemons in chunks, while making sure they confirm getting the information
before sending the next chunk. Also see "Delta Sync".

Gaia
Check Point security operating system that combines the strengths of both
SecurePlatform and IPSO operating systems.

Gaia Clish
The name of the default command line shell in Check Point Gaia operating system. This
is a restrictive shell (role-based administration controls the number of commands
available in the shell).

Gaia Portal
Web interface for Check Point Gaia operating system.

Geo Cluster
A High Availability cluster mode (in R81 and higher versions), where cluster members
are located in different geographical areas (different sites, different cloud availability
zones). This mode supports the configuration of IP addresses from different subnets on
all cluster interfaces, including the Sync interfaces. The Active cluster member inspects
all traffic routed to the cluster and synchronizes the recorded connections to its peer
cluster members. The traffic is not balanced between the cluster members. See "High
Availability".

ClusterXL R81 Administration Guide      |      23

 Glossary

HA not started
Output of the 'cphaprob <flag>' command or 'show cluster <option>' command on the
Cluster Member. This output means that Check Point clustering software is not started
on this Security Gateway (for example, this machine is not a part of a cluster, or
'cphastop' command was run, or some failure occurred that prevented the ClusterXL
product from starting correctly).

High Availability
A redundant cluster mode, where only one Cluster Member (Active member) processes
all the traffic, while other Cluster Members (Standby members) are ready to be promoted
to Active state if the current Active member fails. In the High Availability mode, the
Cluster Virtual IP address (that represents the cluster on that network) is associated: (1)
With physical MAC Address of Active member (2) With virtual MAC Address (see
sk50840). Acronym: HA.

Hotfix
A piece of software installed on top of the current software in order to fix some wrong or
undesired behavior.

HTU
Stands for "HA Time Unit". All internal time in ClusterXL is measured in HTUs (the
times in cluster debug also appear in HTUs). Formula in the Check Point software: 1
HTU = 10 x fwha_timer_base_res = 10 x 10 milliseconds = 100 ms.

Hybrid
Starting in R80.20, on Security Gateways with 40 or more CPU cores, Software Blades
run in the user space (as 'fwk' processes). The Hybrid Mode refers to the state when you
upgrade Cluster Members from R80.10 (or below) to R80.20 (or above). The Hybrid
Mode is the state, in which the upgraded Cluster Members already run their Software
Blades in the user space (as fwk processes), while other Cluster Members still run their
Software Blades in the kernel space (represented by the fw_worker processes). In the
Hybrid Mode, Cluster Members are able to synchronize the required information.

ICA
Internal Certificate Authority. A component on Check Point Management Server that
issues certificates for authentication.

ClusterXL R81 Administration Guide      |      24

 Glossary

Init
State of a Cluster Member in the phase after the boot and until the Full Sync completes.
A Cluster Member in this state does not process any traffic passing through cluster.

Internal Network
Computers and resources protected by the Firewall and accessed by authenticated
users.

IP Tracking
Collecting and saving of Source IP addresses and Source MAC addresses from
incoming IP packets during the probing. IP tracking is a useful for Cluster Members to
determine whether the network connectivity of the Cluster Member is acceptable.

IP Tracking Policy
Internal setting that controls, which IP addresses should be tracked during IP tracking:
(1) Only IP addresses from the subnet of cluster VIP, or from subnet of physical cluster
interface (this is the default) (2) All IP addresses, also outside the cluster subnet.

IPv4
Internet Protocol Version 4 (see RFC 791). A 32-bit number - 4 sets of numbers, each
set can be from 0 - 255. For example, 192.168.2.1.

IPv6
Internet Protocol Version 6 (see RFC 2460 and RFC 3513). 128-bit number - 8 sets of
hexadecimal numbers, each set can be from 0 - ffff. For example,
FEDC:BA98:7654:3210:FEDC:BA98:7654:3210.

Jumbo Hotfix Accumulator

Collection of hotfixes combined into a single package. Acronyms: JHA, JHF.

Link Aggregation
Technology that joins (aggregates) multiple physical interfaces together into one virtual
interface, known as a bond interface. Also known as Interface Bonding, or Interface
Teaming. This increases throughput beyond what a single connection could sustain,
and to provides redundancy in case one of the links should fail.

ClusterXL R81 Administration Guide      |      25

 Glossary

Load Sharing
Also, Load Balancing mode. A redundant cluster mode, where all Cluster Members
process all incoming traffic in parallel. See "Load Sharing Multicast Mode" and "Load
Sharing Unicast Mode". Acronym: LS.

Load Sharing Multicast

Load Sharing Cluster Mode, where all Cluster Members process all traffic in parallel.
Each Cluster Member is assigned the equal load of [ 100% / number_of_members ].
The Cluster Virtual IP address (that represents the cluster on that network) is associated
with Multicast MAC Address 01:00:5E:X:Y:Z (which is generated based on last 3 bytes
of cluster Virtual IP address on that network). A ClusterXL decision algorithm (Decision
Function) on all Cluster Members decides, which Cluster Member should process the
given packet.

Load Sharing Unicast

Load Sharing Cluster Mode, where one Cluster Member (called Pivot) accepts all traffic.
Then, the Pivot member decides to process this traffic, or to forward it to other non-Pivot
Cluster Members. The traffic load is assigned to Cluster Members based on the hard-
coded formula per the value of Pivot_overhead attribute (see sk34668). The Cluster
Virtual IP address (that represents the cluster on that network) is associated with: (1)
Physical MAC Address of Pivot member (2) Virtual MAC Address (see sk50840).

Log
A record of an action that is done by a Software Blade.

Log Server
A dedicated Check Point computer that runs Check Point software to store and process
logs in Security Management Server or Multi-Domain Security Management
environment.

Management High Availability

Deployment and configuration mode of two Check Point Management Servers, in which
they automatically synchronize the management databases with each other. In this
mode, one Management Server is Active, and the other is Standby. Acronyms:
Management HA, MGMT HA.

ClusterXL R81 Administration Guide      |      26

 Glossary

Management Interface
Interface on Gaia computer, through which users connect to Portal or CLI. Interface on a
Gaia Security Gateway or Cluster member, through which Management Server
connects to the Security Gateway or Cluster member.

Management Server
A Check Point Security Management Server or a Multi-Domain Server.

Master
State of a Cluster Member that processes all traffic in cluster configured in VRRP mode.

Multi-Domain Log Server

A computer that runs Check Point software to store and process logs in Multi-Domain
Security Management environment. The Multi-Domain Log Server consists of Domain
Log Servers that store and process logs from Security Gateways that are managed by
the corresponding Domain Management Servers. Acronym: MDLS.

Multi-Domain Security Management

A centralized management solution for large-scale, distributed environments with many
different Domain networks.

Multi-Domain Server
A computer that runs Check Point software to host virtual Security Management Servers
called Domain Management Servers. Acronym: MDS.

Multi-Version Cluster
The Multi-Version Cluster (MVC) mechanism lets you synchronize connections
between cluster members that run different versions. This lets you upgrade to a newer
version without a loss in connectivity and lets you test the new version on some of the
cluster members before you decide to upgrade the rest of the cluster members.

MVC
See "Multi-Version Cluster".

Network Object
Logical representation of every part of corporate topology (physical machine, software
component, IP Address range, service, and so on).

ClusterXL R81 Administration Guide      |      27

 Glossary

Network Objective
Defines how the cluster will configure and monitor an interface - Cluster, Sync,
Cluster+Sync, Monitored Private, Non-Monitored Private. Configured in SmartConsole >
cluster object > 'Topology' pane > 'Network Objective'.

Non-Blocking Mode
Cluster operation mode, in which Cluster Member keeps forwarding all traffic.

Non-Monitored Interface
An interface on a Cluster Member, whose Network Type was set as Private in
SmartConsole, in cluster object.

Non-Pivot
A Cluster Member in the Unicast Load Sharing cluster that receives all packets from the
Pivot Cluster Member.

Non-Sticky Connection
A connection is called non-sticky, if the reply packet returns via a different Cluster
Member, than the original packet (for example, if network administrator has configured
asymmetric routing). In Load Sharing mode, all Cluster Members are Active, and in
Static NAT and encrypted connections, the Source and Destination IP addresses
change. Therefore, Static NAT and encrypted connections through a Load Sharing
cluster may be non-sticky.

Open Server
A physical computer manufactured and distributed by a company, other than Check
Point.

Packet Selection
Distinguishing between different kinds of packets coming from the network, and
selecting, which member should handle a specific packet (Decision Function
mechanism): CCP packet from another member of this cluster; CCP packet from another
cluster or from a Cluster; Member with another version (usually older version of CCP);
Packet is destined directly to this member; Packet is destined to another member of this
cluster; Packet is intended to pass through this Cluster Member; ARP packets.

ClusterXL R81 Administration Guide      |      28

 Glossary

Pingable Host
Some host (that is, some IP address) that Cluster Members can ping during probing
mechanism. Pinging hosts in an interface's subnet is one of the health checks that
ClusterXL mechanism performs. This pingable host will allow the Cluster Members to
determine with more precision what has failed (which interface on which member). On
Sync network, usually, there are no hosts. In such case, if switch supports this, an IP
address should be assigned on the switch (for example, in the relevant VLAN). The IP
address of such pingable host should be assigned per this formula: IP_of_pingable_
host = IP_of_physical_interface_on_member + ~10. Assigning the IP address to
pingable host that is higher than the IP addresses of physical interfaces on the Cluster
Members will give some time to Cluster Members to perform the default health checks.
Example: IP address of physical interface on a given subnet on Member_A is
10.20.30.41; IP address of physical interface on a given subnet on Member_B is
10.20.30.42; IP address of pingable host should be at least 10.20.30.5

Pivot
A Cluster Member in the Unicast Load Sharing cluster that receives all packets. Cluster
Virtual IP addresses are associated with Physical MAC Addresses of this Cluster
Member. This Pivot Cluster Member distributes the traffic between other Non-Pivot
Cluster Members.

Pnote
See "Critical Device".

Preconfigured Mode
Cluster Mode, where cluster membership is enabled on all Cluster Members to be.
However, no policy had been yet installed on any of the Cluster Members - none of
them is actually configured to be primary, secondary, and so on. The cluster cannot
function, if one Cluster Member ‎fails.‎ In ‎this ‎scenario,‎the "preconfigured mode" takes
place. The preconfigured mode also comes into effect when no policy is yet installed,
right after the Cluster Members came up after boot, or when running the 'cphaconf init'
command.

ClusterXL R81 Administration Guide      |      29

 Glossary

Primary Up
ClusterXL in High Availability mode that was configured as Switch to higher priority
Cluster Member in the cluster object in SmartConsole: (1) Each Cluster Member is
given a priority (SmartConsole > cluster object > 'Cluster Members' pane). Cluster
Member with the highest priority appears at the top of the table, and Cluster Member
with the lowest priority appears at the bottom of the table. (2) The Cluster Member with
the highest priority will assume the Active state. (3) If the current Active Cluster Member
with the highest priority (for example, Member_A), fails for some reason, or is rebooted,
then failover occurs between Cluster Members. The Cluster Member with the next
highest priority will be promoted to be Active (for example, Member_B). (4) When the
Cluster Member with the highest priority (Member_A) recovers from a failure, or boots,
then additional failover occurs between Cluster Members. The Cluster Member with the
highest priority (Member_A) will be promoted to Active state (and Member_B will return
to Standby state).

Private Interface
An interface on a Cluster Member, whose Network Type was set as 'Private' in
SmartConsole in cluster object. This interface is not monitored by cluster, and failure on
this interface will not cause any changes in Cluster Member's state.

Probing
If a Cluster Member fails to receive status for another member (does not receive CCP
packets from that member) on a given segment, Cluster Member will probe that segment
in an attempt to illicit a response. The purpose of such probes is to detect the nature of
possible interface failures, and to determine which module has the problem. The
outcome of this probe will determine what action is taken next (change the state of an
interface, or of a Cluster Member).

Problem Notification
See "Critical Device".

Ready
State of a Cluster Member during after initialization and before promotion to the next
required state - Active / Standby / VRRP Master / VRRP Backup (depending on Cluster
Mode). A Cluster Member in this state does not process any traffic passing through
cluster. A member can be stuck in this state due to several reasons - see sk42096.

ClusterXL R81 Administration Guide      |      30

 Glossary

Rule
A set of traffic parameters and other conditions in a Rule Base that cause specified
actions to be taken for a communication session.

Rule Base
Also Rulebase. All rules configured in a given Security Policy.

SecureXL
Check Point product that accelerates IPv4 and IPv6 traffic. Installed on Security
Gateways for significant performance improvements.

Security Gateway
A computer that runs Check Point software to inspect traffic and enforces Security
Policies for connected network resources.

Security Management Server

A computer that runs Check Point software to manage the objects and policies in Check
Point environment.

Security Policy
A collection of rules that control network traffic and enforce organization guidelines for
data protection and access to resources with packet inspection.

Selection
The packet selection mechanism is one of the central and most important components
in the ClusterXL product and State Synchronization infrastructure for 3rd party clustering
solutions. Its main purpose is to decide (to select) correctly what has to be done to the
incoming and outgoing traffic on the Cluster Member. (1) In ClusterXL, the packet is
selected by Cluster Member(s) depending on the cluster mode: In HA modes - by Active
member; In LS Unicast mode - by Pivot member; In LS Multicast mode - by all members.
Then the Cluster Member applies the Decision Function (and the Cluster Correction
Layer). (2) In 3rd party / OPSEC cluster, the 3rd party software selects the packet, and
Check Point software just inspects it (and performs State Synchronization).

ClusterXL R81 Administration Guide      |      31

 Glossary

SIC
Secure Internal Communication. The Check Point proprietary mechanism with which
Check Point computers that run Check Point software authenticate each other over
SSL, for secure communication. This authentication is based on the certificates issued
by the ICA on a Check Point Management Server.

Single Sign-On
A property of access control of multiple related, yet independent, software systems. With
this property, a user logs in with a single ID and password to gain access to a
connected system or systems without using different usernames or passwords, or in
some configurations seamlessly sign on at each system. This is typically accomplished
using the Lightweight Directory Access Protocol (LDAP) and stored LDAP databases
on (directory) servers. Acronym: SSO.

SmartConsole
A Check Point GUI application used to manage Security Policies, monitor products and
events, install updates, provision new devices and appliances, and manage a multi-
domain environment and each domain.

SmartDashboard
A legacy Check Point GUI client used to create and manage the security settings in
R77.30 and lower versions.

SmartUpdate
A legacy Check Point GUI client used to manage licenses and contracts.

Software Blade
A software blade is a security solution based on specific business needs. Each blade is
independent, modular and centrally managed. To extend security, additional blades can
be quickly added.

SSO
See "Single Sign-On".

Standalone
A Check Point computer, on which both the Security Gateway and Security
Management Server products are installed and configured.

ClusterXL R81 Administration Guide      |      32

 Glossary

Standby
State of a Cluster Member that is ready to be promoted to Active state (if the current
Active Cluster Member fails). Applies only to ClusterXL High Availability Mode.

State Synchronization
Technology that synchronizes the relevant information about the current connections
(stored in various kernel tables on Check Point Security Gateways) among all Cluster
Members over Synchronization Network. Due to State Synchronization, the current
connections are not cut off during cluster failover.

Sticky Connection
A connection is called sticky, if all packets are handled by a single Cluster Member (in
High Availability mode, all packets reach the Active Cluster Member, so all connections
are sticky).

Subscribers
User Space processes that are made aware of the current state of the ClusterXL state
machine and other clustering configuration parameters. List of such subscribers can be
obtained by running the 'cphaconf debug_data' command (see sk31499).

Sync Interface
Also, Secured Interface, Trusted Interface. An interface on a Cluster Member, whose
Network Type was set as Sync or Cluster+Sync in SmartConsole in cluster object. This
interface is monitored by cluster, and failure on this interface will cause cluster failover.
This interface is used for State Synchronization between Cluster Members. The use of
more than one Sync Interfaces for redundancy is not supported because the CPU load
will increase significantly due to duplicate tasks performed by all configured
Synchronization Networks. See sk92804.

Synchronization Network
Also, Sync Network, Secured Network, Trusted Network. A set of interfaces on Cluster
Members that were configured as interfaces, over which State Synchronization
information will be passed (as Delta Sync packets ). The use of more than one
Synchronization Network for redundancy is not supported because the CPU load will
increase significantly due to duplicate tasks performed by all configured
Synchronization Networks. See sk92804.

Traffic
Flow of data between network devices.

ClusterXL R81 Administration Guide      |      33

 Glossary

Users
Personnel authorized to use network resources and applications.

VLAN
Virtual Local Area Network. Open servers or appliances connected to a virtual network,
which are not physically connected to the same network.

VLAN Trunk
A connection between two switches that contains multiple VLANs.

VMAC
Virtual MAC address. When this feature is enabled on Cluster Members, all Cluster
Members in High Availability mode and Load Sharing Unicast mode associate the
same Virtual MAC address with Virtual IP address. This allows avoiding issues when
Gratuitous ARP packets sent by cluster during failover are not integrated into ARP
cache table on switches surrounding the cluster. See sk50840.

VSX
Virtual System Extension. Check Point virtual networking solution, hosted on a
computer or cluster with virtual abstractions of Check Point Security Gateways and
other network devices. These Virtual Devices provide the same functionality as their
physical counterparts.

VSX Gateway
Physical server that hosts VSX virtual networks, including all Virtual Devices that
provide the functionality of physical network devices. It holds at least one Virtual
System, which is called VS0.

ClusterXL R81 Administration Guide      |      34

 Introduction to ClusterXL

Introduction to ClusterXL
The Need for Clusters
Security Gateways and VPN connections are business critical devices. The failure of a Security Gateway or
VPN connection can result in the loss of active connections and access to critical data. The Security
Gateway between the organization and the world must remain open under all circumstances.

ClusterXL Solution
ClusterXL is a Check Point software-based cluster solution for Security Gateway redundancy and Load
Sharing. A ClusterXL Security Cluster contains identical Check Point Security Gateways.
n A High Availability Security Cluster ensures Security Gateway and VPN connection redundancy by
providing transparent failover to a backup Security Gateway in the event of failure.
n A Load Sharing Security Cluster provides reliability and also increases performance, as all members
are active.

Item Description

1 Internal network

2 Switch for internal network

3 Security Gateways with ClusterXL Software Blade

4 Switch for external networks

5 Internet

How ClusterXL Works

ClusterXL uses State Synchronization to keep active connections alive and prevent data loss when a
Cluster Member fails. With State Synchronization, each Cluster Member "knows" about connections that
go through other Cluster Members.

ClusterXL R81 Administration Guide      |      35

 Introduction to ClusterXL

ClusterXL uses virtual IP addresses for the cluster itself and unique physical IP and MAC addresses for the
Cluster Members. Virtual IP addresses do not belong to physical interfaces.

Note - This guide contains information only for Security Gateway clusters. For
additional information about the use of ClusterXL with VSX, see the R81 VSX
Administration Guide.

The Cluster Control Protocol

The Cluster Control Protocol (CCP) packets are the glue that links together the members in the Security
Cluster.
CCP traffic is distinct from ordinary network traffic and can be viewed using any network sniffer.
CCP runs on UDP port 8116 between the Cluster Members, and has the following roles:
n It allows Cluster Members to report their own states and learn about the states of other members by
sending keep-alive packets (this only applies to ClusterXL clusters).
n State Synchronization (Delta Sync).
The Check Point CCP is used by all ClusterXL modes.

Important - There is no need to add an explicit rule to the Security Policy Rule Base that
accepts CCP packets.

For more information, see:

n "Configuring the Cluster Control Protocol (CCP) Settings" on page 188
n sk25977

ClusterXL R81 Administration Guide      |      36

 ClusterXL Requirements and Compatibility

ClusterXL Requirements and

Compatibility
In This Section:

Check Point Appliances and Open Servers 37

Supported Number of Cluster Members 38
Hardware Requirements for Cluster Members 38
Software Requirements for Cluster Members 38
VMAC Mode 39
Supported Topologies for Synchronization Network 41
Clock Synchronization in ClusterXL 44
IPv6 Support for ClusterXL 44
Synchronized Cluster Restrictions 44

Check Point Appliances and Open Servers

You can install ClusterXL on Check Point appliances in one of these configurations:
n A Distributed configuration - the Cluster Members and the Security Management Server are
installed on different computers.
n A Full High Availability configuration - the Cluster Members and the Security Management Servers
are installed on the same computers (each computer runs a Standalone configuration).
You can install ClusterXL on Open Servers only in a distributed configuration - the Cluster Members and
the Security Management Server are installed on different computers.
To see the ClusterXL supported platforms, see the R81 Release Notes.
For installation instructions, see the R81 Installation and Upgrade Guide.

ClusterXL R81 Administration Guide      |      37

 ClusterXL Requirements and Compatibility

Supported Number of Cluster Members

n ClusterXL in High Availability mode supports up to 5 Cluster Members.
n ClusterXL in Load Sharing mode supports up to 5 Cluster Members.

Note - Configuring more than 4 members significantly decreases cluster

performance due to amount of Delta Sync.

n VRRP Cluster on Gaia OS supports only 2 Cluster Members (see sk105170).

n Virtual System Load Sharing (VSLS) mode supports up to 13 Cluster Members.

Hardware Requirements for Cluster Members

ClusterXL operation completely relies on internal timers and calculation of internal timeouts, which are
based on hardware clock ticks.
Therefore, in order to avoid unexpected behavior, ClusterXL is supported only between machines with
identical CPU characteristics.

Best Practice - To avoid unexpected fail-overs due to issues with CCP packets on
cluster interfaces, we strongly recommend to pair only identical physical interfaces as
cluster interfaces - even when connecting the Cluster Members via a switch.
For example:
n Intel 82598EB on Member_A with Intel 82598EB on Member_B
n Broadcom NeXtreme on Member_A with Broadcom NeXtreme on Member_B

Note - There is no requirement for throughput of Sync interface to be identical to, or

larger than throughput of traffic interfaces (although, to prevent a possible bottle neck,
a good practice for throughput of Sync interface is to be at least identical to throughput
of traffic interfaces).

Software Requirements for Cluster Members

ClusterXL is supported only between identical operating systems - all Cluster Members must be installed
on the same operating system).
ClusterXL is supported only between identical Check Point software versions - all Cluster Members must
be installed with identical Check Point software, including OS build and hotfixes.
All Check Point software components must be the same on all Cluster Members. Meaning that the same
Software Blades and features must be enabled on all Cluster Members:

ClusterXL R81 Administration Guide      |      38

 ClusterXL Requirements and Compatibility

n SecureXL status on all Cluster Members must be the same (either enabled, or disabled)
n Number of CoreXL Firewall instances on all Cluster Members must be the same

Notes:
l A Cluster Member with a greater number of CoreXL Firewall instances

changes its state to DOWN.

l Fail-over from a Cluster Member to a peer Cluster Member with a greater

number of CoreXL Firewall instances keeps all connections.

l Fail-over from a Cluster Member to a peer Cluster Member with a smaller

number of CoreXL Firewall instances interrupts some connections. The

connections that are interrupted are those that pass through CoreXL
Firewall instances that do not exist on the peer Cluster Member.
n Advanced Dynamic Routing status and configuration on all Cluster Members must be the same
Otherwise, traffic might not be processed as expected and/or state of Cluster Members might change
unexpectedly. In addition, Full Sync will fail.

VMAC Mode
When ClusterXL is configured in High Availability mode or Load Sharing Unicast mode (not Multicast), a
single Cluster Member is associated with the Cluster Virtual IP address. In a High Availability environment,
the single member is the Active member. In a Load Sharing environment, the single member is the Pivot.
After fail-over, the new Active member (or Pivot member) broadcasts a series of Gratuitous ARP Requests
(GARPs). The GARPS associate the Virtual IP address of the cluster with the physical MAC address of the
new Active member or the new Pivot.
When this happens:
n A member with a large number of Static NAT entries can transmit too many GARPs
Switches may not integrate these GARP updates quickly enough into their ARP tables. Switches
continue to send traffic to the physical MAC address of the member that failed. This results in traffic
outage until the switches have fully updated ARP cache tables.
n Network components, such as VoIP phones, ignore GARPs
These components continue to send traffic to the MAC address of the failed member.
To minimize possible traffic outage during a fail-over, configure the cluster to use a virtual MAC address
(VMAC).
By enabling Virtual MAC in ClusterXL High Availability mode, or Load Sharing Unicast mode, all Cluster
Members associate the same Virtual MAC address with all Cluster Virtual Interfaces and the Virtual IP
address. In Virtual MAC mode, the VMAC that is advertised by the Cluster Members (through GARP
Requests) keeps the real MAC address of each member and adds a Virtual MAC address on top of it.
For local connections and sync connections, the real physical MAC address of each Cluster Member is still
associated with its real IP address.

Note - Traffic originating from the Cluster Members will be sent with the VMAC Source address.

You can enable VMAC in SmartConsole, or on the command line. See sk50840.

ClusterXL R81 Administration Guide      |      39

 ClusterXL Requirements and Compatibility

Failover time in a cluster with enabled VMAC mode is shorter than a failover in a cluster that uses a
physical MAC addresses.

Configuring VMAC Mode in SmartConsole

1. Double-click the Cluster object to open its Properties window.

2. On the ClusterXL and VRRP page, select Use Virtual MAC.
3. Click OK.
4. Install the Access Control Policy on this cluster object.

Configuring VMAC Mode on the Command Line

On each Cluster Member, set the same value for the global kernel parameter fwha_vmac_global_
param_enabled.
1. Connect to the command line on each Cluster Member.
2. Log in to the Expert mode.
3. Get the current value of this kernel parameter. Run:

fw ctl get int fwha_vmac_global_param_enabled

4. Set the new value for this kernel parameter temporarily (does not survive reboot). Run:
n To enable VMAC mode:

fw ctl set int fwha_vmac_global_param_enabled 1

n To disable VMAC mode:

fw ctl set int fwha_vmac_global_param_enabled 0

5. Make sure the state of the VMAC mode was changed. Run on each Cluster Member:
n In Gaia Clish:

show cluster members interfaces all

n In the Expert mode:

cphaprob -a if

When VMAC mode is enabled, output of this command shows the VMAC address of each virtual
cluster interface.
6. To set the new value for this kernel parameter permanently:
Follow the instructions in sk26202 to add this line to the
$FWDIR/boot/modules/fwkern.conf file:

fwha_vmac_global_param_enabled=<value>

ClusterXL R81 Administration Guide      |      40

 ClusterXL Requirements and Compatibility

Supported Topologies for Synchronization

Network
Note - The topology numbers below are for convenience only.
Topology 1

1. The Sync interface is a Bond of several physical slave interfaces.

To work with this topology, you can configure the Bond interface in High Availability or Load
Sharing mode.
2. All physical slave interfaces on all Cluster Members connect to the same switch.
3. In this topology, Cluster Members monitor only the link state of the slave interfaces.

Topology 2

1. The Sync interface is a Bond of several physical slave interfaces.

To work with this topology, you can configure the Bond interface in High Availability or Load
Sharing mode.
2. On each Cluster Member, physical slave interfaces of the same Bond connect to different
switches.
3. The switches must connect to each other through a cable.
4. In this topology, Cluster Members monitor only the link state of the slave interfaces.

ClusterXL R81 Administration Guide      |      41

 ClusterXL Requirements and Compatibility

Topology 3 (Enhanced Active/Backup Bond)

1. The Sync interface is a Bond of several physical slave interfaces.

To work with this topology:
a. You must configure the Bond interface in the Active-Backup mode.
b. You must not configure the primary slave explicitly.
2. On each Cluster Member, physical slave interfaces of the same Bond connect to different
switches.
3. The switches do not need to connect to each other through a cable.
4. In this topology, Cluster Members:
n Monitor the link state of the slave interfaces.
n Monitor the path between the Cluster Members, communicate about their connectivity,
and agree on which bond slave interface to use.
n Send Cluster Control Protocol (CCP) packets only on the active slave interface (there is no
CCP broadcast on all the slave interfaces).
n Fail over only if all slave interfaces are down.

Additional information

If a Cluster Member does not receive CCP packets from other Cluster Members (for
example, a cable is disconnected between switches), it sends a dedicated CCP packet
to all other Cluster Members.
This dedicated CCP packet includes a map of all available bond slave interfaces on that
Cluster Member.
When other Cluster Members receive this dedicated CCP packet, they:
a. Compare the received map of available bond slave interfaces to the state of their
own bond slave interfaces.
b. Perform bond internal failover accordingly.

n Fail over to a specific selected slave interface instead of iterating over all available slave
interfaces.

ClusterXL R81 Administration Guide      |      42

 ClusterXL Requirements and Compatibility

Topology 4 (Enhanced Active/Backup Bond)

1. The Sync interface is a Bond of several physical slave interfaces.

To work with this topology:
a. You must configure the Bond interface in the Active-Backup mode.
b. You must not configure the primary slave explicitly.
2. On each Cluster Member, physical slave interfaces of the same Bond connect to different pairs of
switches.
3. These pairs of switches connect to each other (chained together).
4. In this topology, Cluster Members:
n Monitor the link state of the slave interfaces.
n Monitor the path between the Cluster Members, communicate about their connectivity,
and agree on which bond slave interface to use.
n Send Cluster Control Protocol (CCP) packets only on the active slave interface (there is no
CCP broadcast on all the slave interfaces).
n Fail over only if all slave interfaces are down.

Additional information

If a Cluster Member does not receive CCP packets from other Cluster Members (for
example, a cable is disconnected between switches), it sends a dedicated CCP packet
to all other Cluster Members.
This dedicated CCP packet includes a map of all available bond slave interfaces on that
Cluster Member.
When other Cluster Members receive this dedicated CCP packet, they:
a. Compare the received map of available bond slave interfaces to the state of their
own bond slave interfaces.
b. Perform bond internal failover accordingly.

n Fail over to a specific selected slave interface instead of iterating over all available slave
interfaces.

ClusterXL R81 Administration Guide      |      43

 ClusterXL Requirements and Compatibility

Clock Synchronization in ClusterXL

When using ClusterXL, make sure to synchronize the clocks of all of the Cluster Members. You can
synchronize the clocks manually, or using a protocol such as NTP. Features, such as VPN, only function
properly when the clocks of all of the Cluster Members are synchronized.

IPv6 Support for ClusterXL

R81 ClusterXL supports High Availability clusters for IPv6. IPv6 status information is synchronized and the
IPv6 clustering mechanism is activated during failover.

You can define IPv6 addresses for:

n Cluster virtual interfaces
n Member physical interfaces

Limitations:
n IPv6 is not supported for Load Sharing clusters.
n You cannot define IPv6 address for synchronization interfaces.

IPv6 in ClusterXL High Availability:

During failover, a cluster sends gratuitous ARP request packets to update hosts and routers connected to
cluster interfaces. It does this by advertising the new MAC address for the virtual cluster IPv4 addresses.
ClusterXL updates the IPv6 network during failovers. ClusterXL sends Neighbor Advertisement messages
to update the neighbor cache (which is equivalent to the ARP cache in IPv4) by advertising the new MAC
address for the virtual cluster IPv6 address. In addition, ClusterXL will reply to any Neighbor Solicitation
with a target address equal to the Virtual Cluster IPv6 address.

Note - ClusterXL failover event detection is based on IPv4 probing. During state
transition the IPv4 driver instructs the IPv6 driver to reestablish IPv6 network
connectivity to the HA cluster.

Synchronized Cluster Restrictions

These restrictions apply when you synchronize Cluster Members:
n The use of more than one dedicated physical interface for synchronization redundancy is not
supported.
You can use Bonding for synchronization interface redundancy (see "Sync Redundancy" on
page 120).
Synchronization interface redundancy is not supported for VRRP Clusters. See sk92804.
n All Cluster Members must run on identically configured hardware platforms.
n If a Cluster Member goes down, user-authenticated connections through that member are lost.

ClusterXL R81 Administration Guide      |      44

 ClusterXL Requirements and Compatibility

Other Cluster Members cannot restore the connection.

Cluster Members maintain client-authenticated or session-authenticated connections.
The reason for this restriction is that a user space process on Cluster Members maintains the user
authentication state.
Cluster Members cannot synchronize the user space information in the same way as they
synchronize the kernel space information.
Cluster Members save the states of Session Authentication and Client Authentication in kernel
tables, which they synchronize.
n Cluster Members cannot synchronize the connection statutes that use system resources. The
reason is the same as for the user-authenticated connections.
n Accounting information for connections is accumulated on each Cluster Member, sent to the
Management Server, and aggregated.
In the event of a cluster failover, the accounting information that is not yet sent to the Management
Server, is lost.
To minimize this risk, you can reduce the time interval when accounting information is sent.
To do this, in the cluster object > Logs > Additional Logging pane, set a lower value for the Update
Account Log every attribute.

ClusterXL R81 Administration Guide      |      45

 High Availability and Load Sharing Modes in ClusterXL

High Availability and Load Sharing

Modes in ClusterXL
Introduction to High Availability and Load
Sharing modes
ClusterXL is a software-based High Availability and Load Sharing solution that distributes network traffic
between clusters of redundant Security Gateways.
ClusterXL has these High Availability features:
n Transparent failover in case of member failures
n Zero downtime for mission-critical environments (when using State Synchronization)
n Enhanced throughput (in Load Sharing modes)
n Transparent upgrades
All members in the cluster are aware of the connections passing through each of the other members. The
Cluster Members synchronize their connection and status information across a secure synchronization
network.
The glue that binds the members in a ClusterXL cluster is the Cluster Control Protocol (CCP), which is used
to pass synchronization and other information between the Cluster Members.

High Availability
In a High Availability cluster, only one member is active (Active/Standby operation). In the event that the
active Cluster Member becomes unavailable, all connections are re-directed to a designated standby
without interruption. In a synchronized cluster, the standby Cluster Members are updated with the state of
the connections of the Active Cluster Member.
In a High Availability cluster, each member is assigned a priority. The highest priority member serves as
the Security Gateway in normal circumstances. If this member fails, control is passed to the next highest
priority member. If that member fails, control is passed to the next member, and so on.
Upon Security Gateway recovery, you can maintain the current Active Security Gateway (Active Up), or to
change to the highest priority Security Gateway (Primary Up).
ClusterXL High Availability mode supports both IPv4 and IPv6.

ClusterXL R81 Administration Guide      |      46

 High Availability and Load Sharing Modes in ClusterXL

Load Sharing
ClusterXL Load Sharing distributes traffic within a cluster so that the total throughput of multiple members
is increased. In Load Sharing configurations, all functioning members in the cluster are active, and handle
network traffic (Active/Active operation).
If any member in a cluster becomes unreachable, transparent failover occurs to the remaining operational
members in the cluster, thus providing High Availability. All connections are shared between the remaining
Security Gateways without interruption.
ClusterXL Load Sharing modes do not support IPv6.

ClusterXL R81 Administration Guide      |      47

 Example ClusterXL Topology

Example ClusterXL Topology

ClusterXL uses unique physical IP and MAC addresses for each Cluster Member, and a virtual IP
addresses for the cluster itself.
Cluster interface virtual IP addresses do not belong to any real member interface.

Example Diagram
The following diagram illustrates a two-member ClusterXL cluster, showing the cluster Virtual IP addresses
and members physical IP addresses.
This sample deployment is used in many of the examples presented in this chapter.

Item Description

1 Internal network

2 Internal switch (internal cluster IP address 10.10.0.100)

3 Security Gateway - Cluster Member A

3a Virtual interface to the internal network (10.10.0.1)

3b Interface to the Cluster Sync network (10.0.10.1)

3c Virtual interface to the external network (192.168.10.1)

4 Security Gateway - Cluster Member B

4a Virtual interface to the internal network (10.10.0.2)

4b Interface to the Cluster Sync network (10.0.10.2)

4c Virtual interface to the external network (192.168.10.2)

5 External switch (external cluster IP address 192.168.10.100)

6 Internet

Each Cluster Member has three interfaces: one external interface, one internal interface, and one for
synchronization. Cluster Member interfaces facing in each direction are connected via a hub or switch.

ClusterXL R81 Administration Guide      |      48

 Example ClusterXL Topology

All Cluster Member interfaces facing the same direction must be in the same network. For example, there
must not be a router between Cluster Members.
The Management Server can be located anywhere, and connection should be established to either the
internal or external cluster IP addresses.
These sections present ClusterXL configuration concepts shown in the example.

Note - In these examples, RFC 1918 private addresses in the range 192.168.0.0 to
192.168.255.255 are treated as public IP addresses.

Defining the Cluster Member IP Addresses

The guidelines for configuring each Cluster Member are as follows:
All members within the cluster must have at least three interfaces:
n An interface facing the external network (that, for example, faces the Internet).
n An interface facing the internal network.
n An interface used for synchronization.
All interfaces pointing in a certain direction must be on the same network.
For example, in the previous illustration, there are two Cluster Members, Member_A and Member_B. Each
has an interface with an IP address facing the Internet through a hub or a switch. This is the external
interface with IP address 192.168.10.1 on Member_A and IP address 192.168.10.2 on Member_B.

Note - This release presents an option to use only two interfaces per member, one
external and one internal, and to run synchronization over the internal interface. We do
not recommend this configuration. It should be used for backup only. (See
"Synchronizing Connections in the Cluster" on page 70.)

Defining the Cluster Virtual IP Addresses

In the previous illustration, the IP address of the cluster is 192.168.10.100.
The cluster has one external virtual IP address and one internal virtual IP address.
The external IP address is 192.168.10.100, and the internal IP address is 10.10.0.100.

Defining the Synchronization Network

The previous illustration shows a synchronization interface with a unique IP address on each Cluster
Member - IP 10.0.10.1 on Member_A and IP 10.0.10.2 on Member_B.

ClusterXL R81 Administration Guide      |      49

 Example ClusterXL Topology

Configuring Cluster Addresses on Different Subnets

Only one public IP address is required in a ClusterXL cluster, for the virtual cluster interface that faces the
Internet. Physical IP addresses of all Cluster Members can be private.
Configuring different subnets for the cluster IP addresses and the members IP addresses (see "Cluster IP
Addresses on Different Subnets" on page 148) is useful to:
n Configure a cluster to replace one Security Gateway in a pre-configured network, without the need
to allocate new IP addresses to the Cluster Members.
n Allow organizations to use only one public IP address for the ClusterXL Cluster. This saves public IP
addresses.

ClusterXL R81 Administration Guide      |      50

 ClusterXL Mode Considerations

ClusterXL Mode Considerations

Choosing High Availability, Load Sharing, or Active-Active
mode
Which cluster mode to choose depends on the need and requirements of the organization.
n A High Availability cluster mode ensures fail-safe connectivity for the organization.
n A Load Sharing cluster mode ensures fail-safe connectivity for the organization and provides the
additional benefit of increased performance.
n An Active-Active cluster mode supports deployment of Cluster Members in different geographical
areas (in different networks).
See "ClusterXL Mode Comparison" on page 58.

Considerations for the Load Sharing Mode

Load Sharing Multicast mode is an efficient way to handle a high traffic load, because the load is distributed
optimally between all Active Cluster Members.
However, not all switches can be used for Load Sharing Multicast mode. Load Sharing Multicast mode
associates a multicast Cluster MAC addresses with a unicast Cluster Virtual IP addresses. This ensures
that traffic destined for the cluster is received by all Cluster Members.
In response to ARP Request packets for Cluster Virtual IP address, Cluster Members send ARP Replies
that contain a unicast Cluster Virtual IP address and a multicast MAC address. Some switches do not
accept such ARP Replies. For some switches, adding a static ARP entry for the unicast Cluster Virtual IP
address and the multicast MAC address will solve the issue. Other switches do not accept this type of static
ARP entry.
Another consideration is whether your deployment includes networking devices with interfaces operating
in a promiscuous mode. If on the same network segment there exist two such networking devices, and a
ClusterXL in Load Sharing Multicast mode, traffic destined for the cluster that is generated by one of the
networking device could also be processed by the other networking device.
For these cases, use Load Sharing Unicast mode, which does not require the use of multicast MAC
address for the Cluster Virtual IP addresses.

IP Address Migration
If you wish to provide High Availability or Load Sharing to an existing Security Gateways configuration, we
recommend taking the existing IP addresses from the Active Security Gateway, and make these the
Cluster Virtual IP addresses, when feasible. Doing so lets you avoid altering of current IPsec endpoint
identities, as well keep Hide NAT configurations the same in many cases.

ClusterXL R81 Administration Guide      |      51

 ClusterXL Modes

ClusterXL Modes
ClusterXL has several working modes.
This section briefly describes each mode and its relative advantages and disadvantages.
n High Availability Mode
n Load Sharing Multicast Mode
n Load Sharing Unicast Mode
n Active-Active Mode (see "Active-Active Mode in ClusterXL" on page 62)

Note - Many examples in the section refer to the sample deployment shown in the
"Example ClusterXL Topology" on page 48.

High Availability Mode

The ClusterXL High Availability mode provides basic High Availability capabilities in a cluster environment.
This means that the cluster can provide Firewall services even when it encounters a problem, which on a
regular Security Gateway results in a complete loss of connectivity. When combined with Check Point State
Synchronization, ClusterXL High Availability can maintain connections through failover events, in a user-
transparent manner, allowing a flawless connectivity experience. As a result, High Availability provides a
backup mechanism, which organizations can use to reduce the risk of unexpected downtime, especially in
a mission-critical environment (such as money transactions).
To achieve this, ClusterXL High Availability mode designates one of the Cluster Members as the Active
member, while the other Cluster Members remain in Standby mode. The cluster associates the Virtual IP
addresses with the physical MAC addresses of the physical interfaces on the Active member (by matching
the cluster Virtual IP address with the unique MAC address of the appropriate physical interface).
Therefore, all traffic directed at the cluster Virtual IP addresses, is actually routed (and filtered) by the
Active Cluster Member.
The role of each Cluster Member is chosen according to its cluster state and Cluster Member priority.
Cluster Member priorities correspond to the order, in which they appear in the Cluster Members page of
the cluster object in SmartConsole. The top-most Cluster Member has the highest priority. You can modify
this ranking at any time (requires policy installation and causes a failover).
In addition to its role as a Firewall, the Active Cluster Member is also responsible for informing the Standby
Cluster Members of any changes to its cluster state and kernel tables. This keeps the peer Cluster
Members up-to-date with the current traffic passing through the cluster.
Whenever the Active Cluster Member detects a problem that is severe enough to prevent this Cluster
Member from working correctly, failover occurs in the cluster. One of the Standby Cluster Members (the
Cluster Member with the next highest priority) assumes the role of the Active Cluster Member. If State
Synchronization is enabled, the new Active Cluster Member recognizes any open connections and
handles them according to their last known state.
Upon the recovery of a failed former Active Cluster Member with a higher priority, the role of the Active
Cluster Member may or may not be switched back to that Cluster Member. This depends on the cluster
object configuration - Maintain current active Cluster Member, or Switch to higher priority Cluster.
It is important to note that Standby Cluster Members may encounter problems as well. In this case, in the
event of a cluster failover, the Standby Cluster Members are not considered for the role of a new Active
Cluster Member.

ClusterXL R81 Administration Guide      |      52

 ClusterXL Modes

Example

This scenario describes a connection from a Client computer on the external network to a Web server
behind the Cluster (on the internal network).
The cluster of two members is configured in High Availability mode.
Example topology:

[Client on]
[external network]
{IP 192.168.10.78/24}
{DG 192.168.10.100}
|
|
{VIP 192.168.10.100/24}
/ \
| |
{IP 192.168.10.1/24} {IP 192.168.10.2/24}
| |
{Active} {Standby}
[Member1]-----sync-----[Member2]
| |
{IP 10.10.0.1/24} {IP 10.10.0.2/24}
| |
\ /
{VIP 10.10.0.100/24}
|
|
{DG 10.10.0.100}
{IP 10.10.0.34/24}
[Web server on]
[internal network]

Chain of events:
1. The user tries to connect from his Client computer 192.168.10.78 to the Web server 10.10.0.34.
2. The Default Gateway on Client computer is 192.168.10.100 (the cluster Virtual IP address).
3. The Client computer issues an ARP Request for IP 192.168.10.100.
4. The Active Cluster Member (Member1) handles the ARP Request for IP 192.168.10.100.
5. The Active Cluster Member sends the ARP Reply with the MAC address of the external interface,
on which the IP address 192.168.10.1 is configured.
6. The Client computer sends the HTTP request packet to the Active Cluster Member - to the VIP
address 192.168.10.100 and MAC address of the corresponding external interface.
7. The Active Cluster Member handles the HTTP request packet.
8. The Active Cluster Member sends the HTTP request packet to the Web server 10.10.0.34.
9. The Web server handles the HTTP request packet.
10. The Web server generates the HTTP response packet.

ClusterXL R81 Administration Guide      |      53

 ClusterXL Modes

11. The Default Gateway on Web server computer is 10.10.0.100 (the cluster Virtual IP address).
12. The Web server issues an ARP Request for IP 10.10.0.100.
13. The Active Cluster Member handles the ARP Request for IP 10.10.0.100.
14. The Active Cluster Member sends the ARP Reply with the MAC address of the internal interface,
on which the IP address 10.10.0.1 is configured.
15. The Web server sends the HTTP response packet to the Active Cluster Member - to VIP address
10.10.0.100 and MAC address of the corresponding internal interface.
16. The Active Cluster Member handles the HTTP response packet.
17. The Active Cluster Member sends the HTTP response packet to the Client computer
192.168.10.78.
18. From now, all traffic between the Client computer and the Web server is routed through the
Active Cluster Member (Member1).
19. If a failure occurs on the current Active Cluster Member (Member1), the cluster fails over.
20. The Standby Cluster Member (Member2) assumes the role of the Active Cluster Member.
21. The Standby Cluster Member sends Gratuitous ARP Requests to both the 192.168.10.x and the
10.10.0.x networks.
These GARP Requests associate the Cluster Virtual IP addresses with the MAC addresses of the
physical interfaces on the new Active Cluster Member (former Standby Cluster Member):
n Cluster VIP address 192.168.10.100 and MAC address of the corresponding external
interface, on which the IP address 192.168.10.2 is configured.
n Cluster VIP address 10.10.0.100 and MAC address of the corresponding internal
interface, on which the IP address 10.10.0.2 is configured.
22. From now, all traffic between the Client computer and the Web server is routed through the new
Active Cluster Member (Member2).
23. The former Active member (Member1) is now considered to be "down". Upon the recovery of a
former Active Cluster Member, the role of the Active Cluster Member may or may not be switched
back to that Cluster Member, depending on the cluster object configuration.

ClusterXL R81 Administration Guide      |      54

 ClusterXL Modes

Load Sharing Modes

Load Sharing Multicast Mode
Load Sharing lets you distribute network traffic between Cluster Members. In contrast to High Availability,
where only a single member is active at any given time, all Cluster Members in a Load Sharing solution are
active. The Cluster Members decide, which Cluster Member handles which packet. The cluster decision
function performs this task - examining each packet going through the cluster, and determining which
Cluster Member should handle it. As a result, a Load Sharing cluster utilizes all Cluster Members, which
usually leads to an increase in its total throughput.
It is important to understand that ClusterXL Load Sharing, provides a full High Availability solution as well.
When all Cluster Members are Active, traffic is evenly distributed between the Cluster Members. In case of
a failover event, caused by a problem in one of the Cluster Members, the processing of all connections
handled by the faulty member is immediately taken over by the other Cluster Members.
ClusterXL offers two separate Load Sharing solutions: Multicast and Unicast. The two modes differ in the
way Cluster Members receive the packets sent to the cluster. This section describes the Load Sharing
Multicast mode.
The Multicast mechanism, which is provided by the Ethernet network layer, allows several interfaces to be
associated with a single physical (MAC) address. Unlike Broadcast, which binds all interfaces in the same
subnet to a single MAC address, Multicast enables grouping within networks. This means that it is possible
to select the interfaces within a single subnet that will receive packets sent to a given MAC address.
ClusterXL uses the Multicast mechanism to associate the cluster Virtual IP addresses with Multicast MAC
addresses. This makes sure that all packets sent to the cluster, acting as a default gateway on the network,
will reach all Cluster Members. Each Cluster Member then decides, whether it should process the packets
or not. This decision is the core of the Load Sharing mechanism: it has to assure that at least one Cluster
Member will process each packet (so that traffic is not blocked), and that no two Cluster Members will
handle the same packets (so that traffic is not duplicated).
An additional requirement of the decision function, is to route each connection through a Cluster Member,
to ensure that packets that belong to a single connection will be processed by the same Cluster Member.
Unfortunately, this requirement cannot always be enforced, and in some cases, packets of the same
connection will be handled by different Cluster Members. ClusterXL handles these situations using its
State Synchronization mechanism, which mirrors connections on all Cluster Members.

Example

This scenario describes a user logging from the Internet to a Web server behind the cluster that is
configured in Load Sharing Multicast mode.
1. The user requests a connection from 192.168.10.78 (his computer) to 10.10.0.34 (the Web
server).
2. A router on the 192.168.10.x network recognizes 192.168.10.100 (the cluster Virtual IP address)
as the default gateway to the 10.10.0.x network.
3. The router issues an ARP Request for IP 192.168.10.100.
4. One of the Active Cluster Members processes the ARP Request, and responds with the Multicast
MAC assigned to the cluster Virtual IP address of 192.168.10.100.
5. When the Web server responds to the user requests, it recognizes 10.10.0.100 as its default
gateway to the Internet.
6. The Web server issues an ARP Request for IP 10.10.0.100.

ClusterXL R81 Administration Guide      |      55

 ClusterXL Modes

7. One of the Active members processes the ARP Request, and responds with the Multicast MAC
address assigned to the cluster Virtual IP address of 10.10.0.100.
8. All packets sent between the user and the Web server reach every Cluster Member, which
decide whether to process each packet.
9. When a cluster failover occurs, one of the Cluster Members goes down. However, traffic still
reaches all of the Active Cluster Members. Therefore, there is no need to make changes in the
network ARP routing. The only thing that changes, is the cluster decision function, which takes
into account the new state of the Cluster Members.

Load Sharing Unicast Mode

Load Sharing Unicast mode provides a Load Sharing solution adapted to environments, where Multicast
Ethernet cannot operate. In this mode, a single Cluster Member, referred to as Pivot, is associated with the
cluster Virtual IP addresses. This Pivot Cluster Member is the only Cluster Member to receive all packets
sent to the cluster. The Pivot Cluster Member is then responsible for propagating the packets to other
Cluster Members, creating a Load Sharing mechanism. Distribution of packets is performed by applying a
decision function on each packet, the same way it is done in Load Sharing Multicast mode. The difference
is that only one Cluster Member performs this selection: any non-Pivot member that receives a forwarded
packet will handle it, without applying the decision function. Note that non-Pivot Cluster Members are still
active, because they perform routing and Firewall tasks on a share of the traffic (although they do not
perform decisions).
Even though the Pivot Cluster Member is responsible for the decision process, it still acts as a Security
Gateway that processes packets (for example, the decision it makes, can be to handle a packet by itself).
However, since its additional tasks can be time consuming, it is usually assigned a smaller share of the
total load.
When a failure occurs in a non-Pivot member, its handled connections are redistributed between active
non-Pivot Cluster Members, providing the same High Availability capabilities of High Availability and Load
Sharing Multicast. When the Pivot Cluster Member encounters a problem, a regular failover event occurs,
and, in addition, another active non-Pivot member assumes the role of the new Pivot. The Pivot member is
always the active member with the highest priority. This means that when a former Pivot recuperates, it will
retain its previous role.

Example

In this scenario, we use a Load Sharing Unicast cluster as the Security Gateway between the end user
computer and the Web server.
1. The user requests a connection from 192.168.10.78 (his computer) to 10.10.0.34 (the Web
server).
2. A router on the 192.168.10.x network recognizes 192.168.10.100 (the cluster Virtual IP address)
as the default gateway to the 10.10.0.x network.
3. The router issues an ARP Request for IP 192.168.10.100.
4. The Pivot Cluster Member handles the ARP Request, and responds with the MAC address that
corresponds to its own unique IP address of 192.168.10.1.
5. When the Web server responds to the user requests, it recognizes 10.10.0.100 as its default
gateway to the Internet.
6. The Web server issues an ARP Request for IP 10.10.0.100.

ClusterXL R81 Administration Guide      |      56

 ClusterXL Modes

7. The Pivot Cluster Member handles the ARP Request, and responds with the MAC address that
corresponds to its own unique IP address of 10.10.0.1.
8. The user request packet reaches the Pivot Cluster Member on interface 192.168.10.1.
9. The Pivot Cluster Member decides that the second non-Pivot Cluster Member should handle this
packet, and forwards it to 192.168.10.2.
10. The second Cluster Member recognizes the packet as a forwarded packet, and handles it.
11. Further packets are processed by either the Pivot Cluster Member, or forwarded and processed
by the non-Pivot Cluster Member.
12. When a failover occurs from the current Pivot Cluster Member, the second Cluster Member
assumes the role of Pivot.
13. The new Pivot member sends Gratuitous ARP Requests to both the 192.168.10.x and the
10.10.0.x networks. These GARP requests associate the cluster Virtual IP address of
192.168.10.100 with the MAC address that corresponds to the unique IP address of
192.168.10.2, and the cluster Virtual IP address of 10.10.0.100 with the MAC address that
correspond to the unique IP address of 10.10.0.2.
14. Traffic sent to the cluster is now received by the new Pivot Cluster Member, and processed by it
(as it is currently the only Active Cluster Member).
15. When the former Pivot Cluster Member recovers, it re-assumes the role of Pivot, by associating
the cluster Virtual IP addresses with its own unique MAC addresses.

ClusterXL R81 Administration Guide      |      57

 ClusterXL Modes

ClusterXL Mode Comparison

This table summarizes the similarities and differences between the ClusterXL modes.

Load
High Load Sharing Active-
Feature Sharing
Availability Multicast Active
Unicast

High Availability Yes Yes Yes No

Load Sharing No Yes Yes No

Performance Good Excellent Very Good Good

State Synchronization Optional Mandatory Mandatory Optional

Hardware Support All routers Not all routers are All routers All
supported routers

Number of members that deal with 1 N N N

network traffic

Number of members that receive 1 N 1 N

packets from router

How cluster answers ARP requests for Unicast Unicast Unicast N/A
a MAC address

VLAN Tagging Support Yes Yes Yes Yes

ClusterXL R81 Administration Guide      |      58

 Cluster Failover

Cluster Failover
What is Failover?
Failover is a cluster redundancy operation that automatically occurs if a Cluster Member is not functional.
When this occurs, other Cluster Members take over for the failed Cluster Member.
In the High Availability mode:
n If the Active Cluster Member detects that it cannot function as a Cluster Member, it notifies the peer
Standby Cluster Members that it must go down. One of the Standby Cluster Members (with the next
highest priority) will promote itself to the Active state.
n If one of the Standby Cluster Members stops receiving Cluster Control Protocol (CCP) packets from
the current Active Cluster Member, that Standby Cluster Member can assume that the current
Active Cluster Member failed. As a result, one of the Standby Cluster Members (with the next
highest priority) will promote itself to the Active state.
n If you do not use State Synchronization in the cluster, existing connections are interrupted when
cluster failover occurs.
In Load Sharing modes:
n If a Cluster Member detects that it cannot function as a Cluster Member, it notifies the peer Cluster
Members that it must go down. Traffic load will be redistributed between the working Cluster
Members.
n If the Cluster Members stop receiving Cluster Control Protocol (CCP) packets from one of their peer
Cluster Member, those working Cluster Members can assume that their peer Cluster Member failed.
As a result, traffic load will be redistributed between the working Cluster Members.
n Because by design, all Cluster Members are always synchronized, current connections are not
interrupted when cluster failover occurs.
To tell each Cluster Member that the other Cluster Members are alive and functioning, the ClusterXL
Cluster Control Protocol (CCP) maintains a heartbeat between Cluster Members. If after a predefined
time, no CCP packets are received from a Cluster Member, it is assumed that the Cluster Member is down.
As a result, cluster failover can occur.
Note that more than one Cluster Member may encounter a problem that will result in a cluster failover
event. In cases where all Cluster Members encounter such problems, ClusterXL will try to choose a single
Cluster Member to continue operating. The state of the chosen member will be reported as Active(!). This
situation lasts until another Cluster Member fully recovers. For example, if a cross cable connecting the
sync interfaces on Cluster Members malfunctions, both Cluster Members will detect an interface problem.
One of them will change to the Down state, and the other to Active (!) state.

When Does a Failover Occur?

A failover takes place when one of the following occurs in a cluster:
n Any Critical Device reports its state as "problem" (see "Viewing Critical Devices" on page 207).
For example, the "fwd" process failed, or Security Policy is uninstalled on a Cluster Member.
n A Cluster Member does not receive Cluster Control Protocol (CCP) packets from its peer Cluster
Member.
For more on failovers, see sk62570.

ClusterXL R81 Administration Guide      |      59

 Cluster Failover

What Happens When a Cluster Member Recovers?

In the High Availability mode:
n If cluster object is configured as Maintain current active Cluster Member, it means any Cluster
Member that becomes Active, remains Active.
If the Cluster Member with highest priority fails, cluster failover occurs. A Cluster Member with the
next highest priority becomes Active.
If the Cluster Member with highest priority recovers, cluster failover does not occurs again, and that
Cluster Member becomes Standby.
n If cluster object is configured as Switch to higher priority Cluster Member, it means that Cluster
Member with the highest priority always has to be Active.
Cluster Member with the highest priority is the Cluster Member that appears at the top of the list in
Cluster object > Cluster Members pane.
If the Cluster Member with the highest priority fails, cluster failover occurs. A peer Cluster Member in
Standby state, with the next highest priority, becomes Active.
If the Cluster Member with the highest priority recovers, cluster failover occurs again. The Cluster
Member with the highest priority becomes Active again. The Cluster Member with the next highest
priority that was Active, returns to the Standby state.
In the Load Sharing modes:
n When the failed Cluster Member recovers, all connections are redistributed between all Active
Cluster Members.

How a Recovered Cluster Member Obtains the Security

Policy
The Administrator installs the Security Policy on the cluster object, rather than separately on individual
Cluster Members. The policy is automatically installed on all Cluster Members. The policy is sent to the IP
addresses defined in the General Properties page of the cluster member object.
When a failed cluster member recovers, first it tries to fetch a policy from one of the peer Active Cluster
Members. The assumption is that the other Cluster Members have a more up to date policy. If fetching a
policy from peer cluster member fails, the recovered cluster member compares its own local policy to the
policy on its Management Server. If the policy on the Management Server is more up to date than the one
on the recovered cluster member, the policy is fetched from the Management Server. If the cluster
member does not have a local policy, it retrieves one from the Management Server. This ensures that all
Cluster Members use the same policy at any given moment.

ClusterXL R81 Administration Guide      |      60

 Cluster Failover

General Failover Limitations

Some connections may not survive cluster failover:
n Security Servers connections.
n Connections that are handled by the Check Point services, in which the option Synchronize
connections on cluster is disabled.
n Connections initiated by the Cluster Member itself.
n TCP connections handled by the Check Point Active Streaming (CPAS) or Passive Streaming Layer
(PSL) mechanism.
n Connections handled by Software Blades:
l If the IPS Software Blade in the cluster object R77.30 is configured to Prefer connectivity ,
and the Cluster Member that owns the connections is Down, then the connection is accepted
without inspection.
Otherwise, the Cluster Members drop the connection.
l For all other Software Blades:
o If the destination Cluster Member is available, the connection is forwarded to the
Cluster Member that owns the connection.
o If the destination Cluster Member is not available, the Cluster Members drop the
connection.

ClusterXL R81 Administration Guide      |      61

 Active-Active Mode in ClusterXL

Active-Active Mode in ClusterXL

Introduction
R80.40 introduced a new ClusterXL mode called Active-Active.
This mode is designed for a cluster, whose Cluster Members are located in different geographical areas
(different sites, different cloud availability zones).
The IP addresses of the interfaces on each Cluster Member are on different networks (including the Sync
interfaces).
Each Cluster Member inspects all traffic routed to it and synchronizes the recorded connections to its peer
Cluster Members.
The traffic is not balanced between the members.
Example Topology:

Important:
n You can configure the Active-Active mode only on Management Server R80.40
(and higher) and only on ClusterXL R80.40 (and higher).
n The Active-Active mode does not provide Load Sharing of the traffic.
Administrator must monitor the load on each Cluster Member (see "Monitoring
and Troubleshooting Clusters" on page 197).
n Cluster Control Protocol (CCP) Encryption must be enabled, which is the default
(see "Configuring the Cluster Control Protocol (CCP) Settings" on page 188).
n It is possible to configure the Cluster Control Protocol (CCP) to work on Layer 2.
n It is possible to enable Dynamic Routing on each Cluster Member, so they
become routers in the applicable Area or Autonomous System on the site.
In this case, you must enable the Bidirectional Forwarding Detection (BFD - ip-
reachability-detection) in the dynamic routing protocol on each cluster interface
and on the cluster sync interface (Known Limitation PMTR-41292).

ClusterXL R81 Administration Guide      |      62

 Active-Active Mode in ClusterXL

Configuring Active-Active mode

Step Instructions

1 Install one ClusterXL Cluster Member on each site.

See the R81 Installation and Upgrade Guide.
Important:
On all Cluster Members in Active-Active mode, names of interfaces that belong to the
same "side" must be identical.
Examples:
n If you connected the interface eth1 to Switch #A on one Cluster Member, then
you must connect the interface eth1 to Switch #A on all other Cluster
Members.
n If you configured the interface eth2 as a Sync interface on one Cluster
Member, then you must configure the interface eth2 as a Sync on all other
Cluster Members.

2 Optional: On each Cluster Member, enable Dynamic Routing, so they become routers in the
applicable Area or Autonomous System on the site.
Important - In this case, you must enable the Bidirectional Forwarding Detection (BFD
- ip-reachability-detection) in the dynamic routing protocol on each cluster interface
and on the cluster sync interface (Known Limitation PMTR-41292).
See the R81 Gaia Advanced Routing Administration Guide.

3 Connect with SmartConsole to the Management Server.

4 From the left navigation panel, click Gateways & Servers .

5 Create a new Cluster object in one of these ways:

n From the top toolbar, click the New ( ) > Cluster > Cluster.
n In the top left corner, click Objects menu > More object types > Network Object >
Gateways and Servers > Cluster > New Cluster.
n In the top right corner, click Objects Pane > New > More > Network Object >
Gateways and Servers > Cluster > Cluster.
In the Check Point Security Gateway Cluster Creation window, you must click Classic
Mode.

6 From the left tree, click General Properties .

a. In the IPv4 Address field, you must enter the 0.0.0.0 address.
b. On the Network Security tab, you must clear the IPsec VPN.

ClusterXL R81 Administration Guide      |      63

 Active-Active Mode in ClusterXL

Step Instructions

7 From the left tree, click ClusterXL and VRRP.

a. In the Select the cluster mode and configuration section, select Active-Active.
b. In the Tracking section, select the applicable option.
c. Optional: In the Advanced Settings section, select Use State Synchronization.

Best Practice - Enable this setting, so Cluster Members can synchronize the
inspected connections.

8 Click OK to update the cluster object properties with the new cluster mode.

9 Open the cluster object and continue the configuration.

10 From the left tree, click Network Management.

a. From the top, click the Get Interfaces > Get Interfaces With Topology .

Important - On all Cluster Members in Active-Active mode, names of interfaces

that belong to the same "side" must be identical.

b. Select each interface and click Edit.

c. From the left tree, click the General page.
d. In the General section, in the Network Type field, select the applicable type:
n For cluster traffic interfaces, select Cluster.
In the IPv4 field, the dummy IP address 0.0.0.0 / 24 is configured automatically.
Note - SmartConsole requires an IP address configured for each
interface. Cluster Members in the Active-Active mode do not use these
settings.
For cluster synchronization interfaces, select Sync .
n

Notes:
l Check Point cluster supports only one synchronization network. If

redundancy is required, configure a Bond interface.

l In the Network Type field, it is not supported to select

"Cluster+Sync " when you deploy a cluster in a cloud (for

example: AWS, Azure).
n For interfaces that do not pass the traffic between the connected networks, select
Private.
e. In the Member IPs section, make sure the IPv4 address and its Net Mask are correct on
each Cluster Member.
f. In the Topology section:
Make sure the settings are correct in the Leads To and Security Zone fields.
Only these options are supported on cluster interfaces:
n Override > Network defined by routes (this is the default).
n Override > Specific > select the applicable Network object or Network Group
object.

11 Click OK.

12 Publish the SmartConsole session.

ClusterXL R81 Administration Guide      |      64

 Active-Active Mode in ClusterXL

Step Instructions

13 Configure and install the applicable Access Control Policy.

14 Configure and install the applicable Threat Prevention Policy.

15 Examine the cluster state.

See "Viewing Cluster State" on page 203.
Example
Member1> show cluster state

Cluster Mode: Active Active with IGMP Membership

ID Unique Address Assigned Load State Name

1 (local) 11.22.33.245 N/A ACTIVE Member1

2 11.22.33.246 N/A ACTIVE Member2

Active PNOTEs: None

... ... ...

Member1>

ClusterXL R81 Administration Guide      |      65

 Active-Active Mode in ClusterXL

Dynamic Routing Failover

By design, a Cluster Member changes its state to DOWN in these cases:
n If there is an issue with the Sync interface (interface state or interface link).
In this case, the Critical Device Interfaces Active Check on the Cluster Member reports its state as
"problem".
n If you run the command "clusterXL_admin down" (see "The clusterXL_admin Script" on
page 273).
In this case, the Critical Device admin_down on the Cluster Member reports its state as "problem".
When the cluster state of a Cluster Member is DOWN, it stops processing the dynamic routing traffic to
force the next hop router to update its routing tables. As a result, there may be a network outage, because
it takes time for dynamic routing protocols to update their routing tables and propagate the changes.

ClusterXL R81 Administration Guide      |      66

 Active-Active Mode in ClusterXL

Note - If it is necessary that Cluster Members change their cluster state because of
other Critical Devices, you must manually configure this behavior.
Procedure

1. Connect to the command line on the Cluster Member.

2. Log in to the Expert mode.
3. Get the list of all Critical Devices:
cphaprob -l list
4. Copy the names of the applicable Critical Devices (case sensitive)
5. Back up the current $FWDIR/boot/modules/fwkern.conf file:
cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}
6. Edit the current $FWDIR/boot/modules/fwkern.conf file:
vi $FWDIR/boot/modules/fwkern.conf
7. Add this line:
fwha_disable_member_state_
pnotelist="<
NameOfCriticalDevice1>,<NameOfCriticalDevice2>,..."
Example:
fwha_disable_member_state_pnotelist="ted,Interface
Active Check,cvpnd"
Important:
n This configuration files do not support space characters,
tabulation characters, and comments (lines that contain the #
character).
n You must write the names of Critical Devices exactly as they
appear in the output of the "cphaprob -l list" command
(in the line "Device Name:").
n Names of Critical Devices must be separated by comma
(without spaces). Do not enclose the names of Critical Devices
in quotes.
8. Save the changes in the file and exit the editor.
9. Reboot the Cluster Member.
10. If you cannot reboot at this time, then configure this behavior temporarily with
this command:
fw ctl set str fwha_disable_member_state_pnotelist
"<
NameOfCriticalDevice1>,<NameOfCriticalDevice2>,..."
Example:
fw ctl set str fwha_disable_member_state_pnotelist
"ted,Interface Active Check,cvpnd"
Important:
n You must write the names of Critical Devices exactly as they
appear in the output of the "cphaprob -l list" command
(in the line "Device Name:").
n Names of Critical Devices must be separated by comma
(without spaces).

ClusterXL R81 Administration Guide      |      67

 Active-Active Mode in ClusterXL

11. Make sure the Cluster Member applied the new configuration:
fw ctl get str fwha_disable_member_state_pnotelist

ClusterXL R81 Administration Guide      |      68

 Active-Active Mode in ClusterXL

Limitations
Feature or
Note
Configuration

Supported Software Blades Only these:

n Firewall
n IPS

Number of Cluster Only one Cluster Member is supported on each site.

Members on each site

Number of Cluster Up to five Cluster Members are supported in a cluster (see "ClusterXL
Members in cluster Requirements and Compatibility" on page 37).

Names of interfaces on On all Cluster Members in Active-Active mode, names of interfaces that
Cluster Members belong to the same "side" must be identical.
Examples:
n If you connected the interface eth1 to Switch #A on one Cluster
Member, then you must connect the interface eth1 to Switch #A
on all other Cluster Members.
n If you configured the interface eth2 as a Sync interface on one
Cluster Member, then you must configure the interface eth2 as a
Sync on all other Cluster Members.

Multi Portal All multi-portals are not supported (Mobile Access Portal, Identity
Awareness Captive Portal, Data Loss Prevention Portal, and so on).

NAT NAT on the IP addresses that belong to cluster interfaces is not

supported (because it does not survive cluster failover).

Network Type configuration In the cluster object properties, go the Network Management page,
of cluster interfaces in select a cluster interface and click Edit.
SmartConsole In the Network Type field, it is not supported to select "Cluster+Sync "
when you deploy a cluster in a cloud (for example: AWS, Azure).

Topology configuration of In the cluster object properties, go the Network Management page,
cluster interfaces in select a cluster interface and click Edit.
SmartConsole In the Topology section, only these options are supported for cluster
interfaces:
n Override > Network defined by routes (this is the default).
n Override > Specific > select the applicable Network object or
Network Group object.

ClusterXL R81 Administration Guide      |      69

 Synchronizing Connections in the Cluster

Synchronizing Connections in the

Cluster
A failure of a Firewall results in an immediate loss of active connections in and out of the organization.
Many of these connections, such as financial transactions, may be mission critical, and losing them will
result in the loss of critical data. ClusterXL supplies an infrastructure that ensures that no data is lost in
case of a failure, by making sure each Cluster Member is aware of the connections going through the other
members. Passing information about connections and other Security Gateway states between the Cluster
Members is called State Synchronization.
Every IP-based service (including TCP and UDP) recognized by the Security Gateway, is synchronized.
Members of a ClusterXL in Load Sharing mode must be synchronized.
Members of a ClusterXL in High Availability mode do not have to be synchronized. Although, if they are not
synchronized, current connections are interrupted during cluster failover.

ClusterXL R81 Administration Guide      |      70

 The Synchronization Network

The Synchronization Network

The Synchronization Network is used to transfer synchronization information about connections and other
Security Gateway states between Cluster Members.
The synchronization network carries the most sensitive Security Policy information in the organization.
Therefore, it is critical that you protect it against both malicious and unintentional threats.
We recommend that you secure the synchronization interfaces using one of these strategies:
n Enable the CCP Encryption (this is the default) on the Cluster Members (see "Configuring the
Cluster Control Protocol (CCP) Settings" on page 188).
n Use a dedicated synchronization network.
n Connecting the physical network interfaces of the Cluster Members directly using a cross-cable. In a
cluster with three or more members, use a dedicated hub or switch.

Notes:
n See "Supported Topologies for Synchronization Network" on page 41.
n You can synchronize members across a WAN. See "Synchronizing Clusters on
a Wide Area Network" on page 82.
n In ClusterXL, the synchronization network is supported on the lowest VLAN tag
of a VLAN interface.
For example, if three VLANs with tags 10, 20 and 30 are configured on interface
eth1, only interface eth1.10 may be used for synchronization.

ClusterXL R81 Administration Guide      |      71

 How State Synchronization Works

How State Synchronization Works

Synchronization works in two modes:
n Full Sync transfers all Security Gateway kernel table information from one Cluster Member to
another.
The fwd daemon handles the Full Sync using an encrypted TCP connection on port 256.
n Delta Sync transfers the changes in the kernel tables between Cluster Members.
The Security Gateway kernel handles the Delta Sync using UDP connections on port 8116.
Full Sync is used for initial transfers of state information, when a Cluster Member joins the cluster. If a
Cluster Member is brought up after being down, it performs the Full Sync with the Active peer Cluster
Member(s). After all Cluster Members are synchronized, only updates are transferred using the Delta
Sync, because the Delta Sync is quicker than the Full Sync.
State Synchronization traffic typically makes up around 90% of all Cluster Control Protocol (CCP) traffic.
Cluster Members distinguish the State Synchronization packets from the rest of CCP traffic based on the
opcode in the UDP data header.

ClusterXL R81 Administration Guide      |      72

 Configuring Services to Synchronize After a Delay

Configuring Services to Synchronize After a

Delay
Some TCP services (for example, HTTP) are characterized by connections with a very short duration.
There is no point to synchronize these connections, because every synchronized connection consumes
resources on Cluster Members, and the connection is likely to have finished by the time a cluster failover
occurs.
For short-lived services, you can use the Delayed Notifications feature to delay telling the Cluster Member
about a connection, so that the connection is only synchronized, if it still exists X seconds after the
connection was initiated. The Delayed Notifications feature requires SecureXL to be enabled on all Cluster
Members (this is the default).

Procedure:
1. In SmartConsole, click Objects > Object Explorer.
2. In the left tree, click the small arrow on the left of the Services to expand this category
3. In the left tree, select TCP.
4. Search for the applicable TCP service.
5. Double-click the applicable TCP service.
6. In the TCP service properties window, click Advanced page.
7. At the top, select Override default settings .
On Domain Management Server: select Override global domain settings .
8. At the bottom, in the Cluster and synchronization section, select Start synchronizing and enter
the applicable value.

Important - This change applies to all policies that use this service.

9. Click OK.
10. Close the Object Explorer.
11. Publish the SmartConsole session.
12. Install the Access Control Policy on the cluster object.

Note - The Delayed Notifications setting in the service object is ignored, if Connection
Templates are not offloaded by the Firewall to SecureXL. For additional information
about the Connection Templates, see the R81 Performance Tuning Administration
Guide.

ClusterXL R81 Administration Guide      |      73

 Configuring Services not to Synchronize

Configuring Services not to Synchronize

Synchronization of connections incurs a performance cost. Not all connections that go through a cluster
must be synchronized:
n Protocols that run solely between Cluster Members need not be synchronized. Although, you can
synchronize them, you go not gain any benefit. This synchronization information does not help
during a cluster failover.
n You can decide not to synchronize TCP, UDP and other service types. By default, Cluster Members
synchronize all these services.
n The VRRP and the IGMP protocols are not synchronized by default (but you can choose to turn on
synchronization for these protocols).
n Broadcast and multicast connections are not, and cannot be, synchronized.
You may choose not to synchronize a service if these conditions are true:
n A significant amount of traffic goes through the cluster. Not synchronizing the service reduces the
amount of synchronization traffic, and so enhances cluster performance.
n The service typically opens short connections, whose loss may not be noticed. DNS (over UDP) and
HTTP are typically responsible for most connections, frequently have short life, and inherent
recoverability in the application level. Services that open long connections, such as FTP, should
always be synchronized.
n Configurations that ensure bi-directional stickiness for all connections, do not require
synchronization to operate (only to maintain High Availability). Such configurations include:
l Any cluster in High Availability mode (for example, ClusterXL High Availability, or VRRP on
Gaia).
l ClusterXL in a Load Sharing mode with clear connections (no VPN, or Static NAT).
l VPN and Static NAT connections passing through a ClusterXL cluster in a Load Sharing mode
(either multicast, or unicast) may not maintain bi-directional stickiness. State Synchronization
must be turned on for such environments.
You can have a synchronized service and a non-synchronized definition of a service, and use them
selectively in the Rule Base. For more information, see the R81 Security Management Administration
Guide.

ClusterXL R81 Administration Guide      |      74

 Configuring Services not to Synchronize

To configure a service not to synchronize in a cluster:

1. In SmartConsole, click Objects > Object Explorer.
2. In the left tree, select Services .
3. Double-click the applicable existing synchronized service, for which you need to create a non-
synchronized counterpart service.
4. Write down all the settings from both the General and Advanced pages.
5. Click OK.
6. Click New > Service > > select the applicable service type.
7. Enter the applicable name that distinguishes the new non-synchronized counterpart service from
the existing synchronized service.
8. On the General page, configure the same settings as in the existing synchronized service.
9. On the Advanced page:
a. Configure the same settings as in the existing synchronized service.
b. In the Cluster and synchronization section, clear Synchronize connections if State
Synchronization is enabled on the cluster.

Important - This change applies to all policies that use this service.

10. Click OK.

11. Close the Object Explorer.
12. Use the synchronized service and the non-synchronized counterpart service in the applicable rules
in the applicable Access Control Policies.
13. Publish the SmartConsole session.
14. Install the Access Control Policy on the cluster object.

ClusterXL R81 Administration Guide      |      75

 Sticky Connections

Sticky Connections
In This Section:

Introduction to Sticky Connections 76

VPN Tunnels with 3rd Party Peers and Load Sharing 76
Third-Party Gateways in Hub and Spoke VPN Deployments 77
Configuring a Third-Party Gateway in a Hub and Spoke VPN Deployment 78

Introduction to Sticky Connections

A connection is considered sticky , when all of its packets are handled, in either direction, by a single
Cluster Member.
This is the case in High Availability mode, where all connections are routed through the same Cluster
Member, and hence, are sticky.
This is also the case in Load Sharing mode, when there are no VPN peers, Static NAT rules, or SIP traffic.
In Load Sharing mode, there are cases, where it is necessary to ensure that a connection that starts on a
specific Cluster Member will continue to be processed by the same Cluster Member in both directions.
Certain connections can be made sticky by enabling the Sticky Decision Function in the cluster object in
SmartConsole.
The Check Point Cluster Correction Layer (CCL) deals with asymmetric connections in Check Point
cluster.

VPN Tunnels with 3rd Party Peers and Load Sharing

Check Point provides interoperability with third-party vendor gateways by enabling them to peer with
Check Point Security Gateways. A special case occurs when certain third-party peers (for example,
Microsoft LT2P, Cisco gateways) attempt to establish VPN tunnels with ClusterXL in Load Sharing mode.
These VPN peers are limited in their ability to store SAs, which means that a VPN session that begins on
one Cluster Member and, due to Load Sharing, is routed on the return trip through another Cluster
Member, is unrecognized and dropped.

Item Description

1 Internal network

ClusterXL R81 Administration Guide      |      76

 Sticky Connections

Item Description

2 Switch for internal network

3 ClusterXL in Load Sharing mode - Cluster Members "A" and "B"

4 Switch for external networks

5 Internet

6 3rd party peer VPN Gateway

7 3rd party peer laptop with VPN client

In this scenario:
n A third-party peer (gateway or client) attempts to create a VPN tunnel.
n Cluster Members "A" and "B" belong to a ClusterXL in Load Sharing mode.
The third-party peers, lacking the ability to store more than one set of SAs, cannot negotiate a VPN tunnel
with multiple Cluster Members, and therefore the Cluster Member cannot complete the routing
transaction.
This issue is resolved for certain third-party peers or gateways that can save only one set of SAs by making
the connection sticky. The Cluster Correction Layer (CCL) makes sure that a single Cluster Member
processes all VPN sessions, initiated by the same third-party gateway.

Third-Party Gateways in Hub and Spoke VPN Deployments

Another case, where Load Sharing mode requires the connection stickiness, which the Cluster Correction
Layer (CCL) provides, is when integrating certain third-party gateways into a Hub and Spoke deployment.
Without the ability to store more than one set of SAs, a third-party gateway must maintain its VPN tunnels
on a single Cluster Member in order to avoid duplicate SAs.

Item Description

1 Security Gateway - Cluster Member A

2 Security Gateway - Cluster Member B

3 Switch for external networks

ClusterXL R81 Administration Guide      |      77

 Sticky Connections

Item Description

4 Internet

5 Gateway - Spoke A

6 Gateway - Spoke B

In this sample deployment:

n The intent of this deployment is to enable hosts that reside behind Spoke A to communicate with
hosts behind Spoke B.
n The ClusterXL in Load Sharing mode, is composed of Cluster Members "A" and "B", and serves as a
VPN Hub.
n Spoke A is a third-party gateway, and is connected by a VPN tunnel that passes through the VPN
Hub to Spoke B.
n Spoke B can be either another third-party gateway, or a Check Point Security Gateway.
Spokes A and B must be set to always communicate using the same Cluster Member. The Cluster
Correction Layer (CCL) solves half of this problem, in that a single Cluster Member processes all VPN
sessions initiated by either third-party gateway.
To make sure that all communications between Spokes A and B are always using the same Cluster
Member, you must make some changes to the applicable user.def file (see sk98239). This second step
ensures that both third-party gateways always connect to the same Cluster Member.

Configuring a Third-Party Gateway in a Hub and Spoke VPN

Deployment
To configure a third-party gateway as a spoke in a Hub and Spoke VPN deployment, perform the following
on the Management Server:
1. Create a Tunnel Group to handle traffic from specific peers. Edit the applicable user.def file (see
sk98239), and add a line similar to the following:

all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>,

<20.20.20.1;1>};

The elements of this configuration are as follows:

n all - All cluster interfaces
n member1,member2 - Names of Cluster Members in SmartConsole
n vpn_sticky_gws - Table name
n 10.10.10.1 - IP address of Spoke A
n 20.20.20.1 - IP address of Spoke B
n ;1 - Tunnel Group Identifier, which indicates that the traffic from these IP addresses should be
handled by the same Cluster Member

ClusterXL R81 Administration Guide      |      78

 Sticky Connections

2. Other VPN peers can be added to the Tunnel Group by including their IP addresses in the same
format as shown above. To continue with the example above, adding Spoke C would look like this:

all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>,

<20.20.20.1;1>, <30.30.30.1;1>};

The Tunnel Group Identifier ;1 stays the same, which means that the listed peers will always connect
through the same Cluster Member.

Note - More tunnel groups than Cluster Members may be defined.

This procedure turns off Load Sharing for the affected connections. If the implementation is to
connect multiple sets of third-party gateways one to another, a form of Load Sharing can be
accomplished by setting Security Gateway pairs to work in tandem with specific Cluster Members.
For instance, to set up a connection between two other spokes (C and D), simply add their IP
addresses to the line and replace the Tunnel Group Identifier ;1 with ;2. The line would then look
something like this:

all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>,

<20.20.20.1;1>, <192.168.15.5;2>, <192.168.1.4;2>,};

Note that there are now two peer identifiers: ;1 and ;2. Spokes A and B will now connect through one
Cluster Member, and Spokes C and D through another.

Note - The tunnel groups are shared between active Cluster Members. In case
of a change in cluster state (for example, failover or Cluster Member
attach/detach), the reassignment is performed according to the new state.

ClusterXL R81 Administration Guide      |      79

 Non-Sticky Connections

Non-Sticky Connections
In This Section:

Non-Sticky Connection Example: TCP 3-Way Handshake 80

Synchronizing Non-Sticky Connections 81

A connection is called sticky if a single Cluster Member handles all packets of that connection. In a non-
sticky connection, the response packet of a connection returns through a different Cluster Member than
the original request packet.
The cluster synchronization mechanism knows how to handle non-sticky connections properly. In a non-
sticky connection, a Cluster Member can receive an out-of-state packet, which Firewall normally drops
because it poses a security risk.
In Load Sharing configurations, all Cluster Members are active. In Static NAT and encrypted connections,
the source and destination IP addresses change. Therefore, Static NAT and encrypted connections
through a Load Sharing cluster may be non-sticky. Non-stickiness may also occur with Hide NAT, but
ClusterXL has a mechanism to make it sticky.
In High Availability configurations, all packets reach the Active member, so all connections are sticky. If
failover occurs during connection establishment, the connection is lost, but synchronization can be
performed later.
If the other members do not know about a non-sticky connection, the packet will be out-of-state, and the
connection will be dropped for security reasons. However, the Synchronization mechanism knows how to
inform other members of the connection. The Synchronization mechanism thereby prevents out-of-state
packets in valid, but non-sticky connections, so that these non-sticky connections are allowed.
Non-sticky connections will also occur if the network Administrator has configured asymmetric routing,
where a reply packet returns through a different Security Gateway than the original packet.

Non-Sticky Connection Example: TCP 3-Way Handshake

The 3-way handshake that initiates all TCP connections can very commonly lead to a non-sticky (often
called asymmetric routing) connection. This diagram shows a sample scenario:

Item Description

1 Client

2 Security Gateway - Cluster Member A

ClusterXL R81 Administration Guide      |      80

 Non-Sticky Connections

Item Description

3 Security Gateway - Cluster Member B

4 Server

The client initiates a connection by sending a SYN packet to the server. The SYN passes through Cluster
Member A, but the SYN-ACK reply returns through Cluster Member B. This is a non-sticky connection,
because the reply packet returns through a different Security Gateway than the original packet.
The synchronization network notifies Cluster Member B. If Cluster Member B is updated before the SYN-
ACK packet sent by the server reaches it, the connection is handled normally. If, however, synchronization
is delayed, and the SYN-ACK packet is received on Cluster Member B before the SYN flag has been
updated, then the Security Gateway treats the SYN-ACK packet as out-of-state, and drops the connection.
You can configure enhanced 3-Way TCP Handshake enforcement to address this issue (see "Enhanced
3-Way TCP Handshake Enforcement" on page 146).

Synchronizing Non-Sticky Connections

The synchronization mechanism prevents out-of-state packets in valid, but non-sticky connections. The
way it does this is best illustrated with reference to the 3-way handshake that initiates all TCP data
connections. The 3-way handshake proceeds as follows:
1. SYN (client to server)
2. SYN-ACK (server to client)
3. ACK (client to server)
4. Data (client to server)
To prevent out-of-state packets, the following sequence (called "Flush and Ack") occurs
1. Cluster Member receives first packet (SYN) of a connection.
2. Suspects that it is non-sticky.
3. Hold the SYN packet.
4. Send the pending synchronization updates to all Cluster Members (including all changes relating to
this packet).
5. Wait for all the other Cluster Members to acknowledge the information in the sync packet.
6. Release held SYN packet.
7. All Cluster Members are ready for the SYN-ACK.

ClusterXL R81 Administration Guide      |      81

 Non-Sticky Connections

Synchronizing Clusters on a Wide Area Network

Organizations sometimes need to locate Cluster Members in geographical locations that are distant from
each other.
A typical example is a replicated Data Center, whose locations are widely separated for disaster recovery
purposes.
In such a configuration, it is clearly impractical to use a cross cable for the synchronization network.
The synchronization network can be spread over remote sites, which makes it easier to deploy
geographically distributed clustering.
There are two limitations to this capability:
1. The synchronization network must guarantee no more than 100ms latency and no more than 5%
packet loss.
2. The synchronization network may include only Layer 2 networking devices - switches and hubs.
No Layer 3 routers are allowed on the synchronization network, because routers drop Cluster
Control Protocol (CCP) packets.

Note - The Active-Active mode also supports Layer 3 (see "Active-Active Mode
in ClusterXL" on page 62).

You can monitor and troubleshoot geographically distributed clusters using the command line interface.

Synchronized Cluster Restrictions

These restrictions apply when you synchronize Cluster Members:
n The use of more than one dedicated physical interface for synchronization redundancy is not
supported.
You can use Bonding for synchronization interface redundancy (see "Sync Redundancy" on
page 120).
Synchronization interface redundancy is not supported for VRRP Clusters. See sk92804.
n All Cluster Members must run on identically configured hardware platforms.
n If a Cluster Member goes down, user-authenticated connections through that member are lost.
Other Cluster Members cannot restore the connection.
Cluster Members maintain client-authenticated or session-authenticated connections.
The reason for this restriction is that a user space process on Cluster Members maintains the user
authentication state.
Cluster Members cannot synchronize the user space information in the same way as they
synchronize the kernel space information.
Cluster Members save the states of Session Authentication and Client Authentication in kernel
tables, which they synchronize.

ClusterXL R81 Administration Guide      |      82

 Non-Sticky Connections

n Cluster Members cannot synchronize the connection statutes that use system resources. The
reason is the same as for the user-authenticated connections.
n Accounting information for connections is accumulated on each Cluster Member, sent to the
Management Server, and aggregated.
In the event of a cluster failover, the accounting information that is not yet sent to the Management
Server, is lost.
To minimize this risk, you can reduce the time interval when accounting information is sent.
To do this, in the cluster object > Logs > Additional Logging pane, set a lower value for the Update
Account Log every attribute.

ClusterXL R81 Administration Guide      |      83

 Non-Sticky Connections

Synchronized Cluster Restrictions

These restrictions apply when you synchronize Cluster Members:
n The use of more than one dedicated physical interface for synchronization redundancy is not
supported.
You can use Bonding for synchronization interface redundancy (see "Sync Redundancy" on
page 120).
Synchronization interface redundancy is not supported for VRRP Clusters. See sk92804.
n All Cluster Members must run on identically configured hardware platforms.
n If a Cluster Member goes down, user-authenticated connections through that member are lost.
Other Cluster Members cannot restore the connection.
Cluster Members maintain client-authenticated or session-authenticated connections.
The reason for this restriction is that a user space process on Cluster Members maintains the user
authentication state.
Cluster Members cannot synchronize the user space information in the same way as they
synchronize the kernel space information.
Cluster Members save the states of Session Authentication and Client Authentication in kernel
tables, which they synchronize.
n Cluster Members cannot synchronize the connection statutes that use system resources. The
reason is the same as for the user-authenticated connections.
n Accounting information for connections is accumulated on each Cluster Member, sent to the
Management Server, and aggregated.
In the event of a cluster failover, the accounting information that is not yet sent to the Management
Server, is lost.
To minimize this risk, you can reduce the time interval when accounting information is sent.
To do this, in the cluster object > Logs > Additional Logging pane, set a lower value for the Update
Account Log every attribute.

ClusterXL R81 Administration Guide      |      84

 Configuring ClusterXL

Configuring ClusterXL
This procedure describes how to configure the Load Sharing Multicast, Load Sharing Unicast, and High
Availability modes from scratch.
Their configuration is identical, apart from the mode selection in SmartConsole Cluster object or Cluster
creation wizard.

Installing Cluster Members

Step Instructions

1 See "ClusterXL Requirements and Compatibility" on page 37.

2 See R81 Installation and Upgrade Guide.

Configuring Routing for Client Computers

Example topology:
[internal network 10.10.2.0/24] --- (VIP 10.10.2.100/24) [Cluster] (VIP 192.168.2.100/24) --- [external
network 192.168.2.0/24]

To configure routing for client computers:

1. Computers on the internal network 10.10.2.0/24 must be configured with Default Gateway IP
10.10.2.100
2. Computers on the external network 192.168.2.0/24 must be configured with Default Gateway IP
192.168.2.100
3. For Proxy ARP configuration, see sk30197.
4. In addition, see "Cluster IP Addresses on Different Subnets" on page 148.

ClusterXL R81 Administration Guide      |      85

 Configuring the Cluster Control Protocol (CCP) Settings

Configuring the Cluster Control Protocol (CCP)

Settings

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
Cluster Members configure the Cluster Control Protocol (CCP) mode automatically.

Important - In R81, the CCP always runs in the unicast mode.

You can configure the Cluster Control Protocol (CCP) Encryption on the Cluster Members.
See "Viewing the Cluster Control Protocol (CCP) Settings" on page 240.

Syntax for configuring the Cluster Control Protocol (CCP) Encryption

Shell Command

Gaia Clish set cluster member ccpenc {off | on}

Expert mode cphaconf ccp_encrypt {off | on}

cphaconf ccp_encrypt_key <Key String>

ClusterXL R81 Administration Guide      |      86

 Configuring the Cluster Object and Members

Configuring the Cluster Object and Members

You can use one of these procedures to define a cluster object and its members:
n Simple Mode (Wizard) - Lets you quickly create a new cluster and configure some basic cluster
properties:
l Cluster properties and Virtual IP addresses
l Properties and Topology of Cluster Members
l Synchronization interfaces and IP addresses
See "Using the Wizard Mode in SmartConsole" below.
n Classic Mode - Opens the Cluster Gateway Properties window, where you manually create a
cluster and configure its properties.
See "Using the Classic Mode in SmartConsole" on page 91.
The Cluster Gateway Properties window lets you:
n Enable and configure Software Blades for the cluster
n Configure other cluster properties that you cannot configure with the wizards
n Change the properties of an existing cluster
See "Changing the Settings of Cluster Object in SmartConsole" on page 95.

Using the Wizard Mode in SmartConsole

This version includes two wizards:
n Check Point Appliances and Open Servers
n Check Point Small Office Appliances

Wizard for Check Point Appliances or Open Servers

The Cluster Wizard is recommended for all Check Point Appliances (except models lower than 3000
series), and for Open Server platforms.

To create a new cluster using Wizard Mode:

Step Instructions

1 In SmartConsole, click Objects menu > More object types > Network Object > Gateways
and Servers > Cluster > New Cluster.

2 In Check Point Security Gateway Cluster Creation window, click Wizard Mode.

ClusterXL R81 Administration Guide      |      87

 Configuring the Cluster Object and Members

Step Instructions

3 In the Cluster General Properties window:

a. In the Cluster Name field, enter unique name for the cluster object.
b. In the Cluster IPv4 Address , enter the unique Cluster Virtual IPv4 addresses for this
cluster.
This is the main IPv4 address of the cluster object.
c. In the Cluster IPv6 Address , enter the unique Cluster Virtual IPv6 addresses for this
cluster.
This is the main IPv6 address of the cluster object.

Important - You must define a corresponding IPv4 address for every IPv6
address. This release does not support pure IPv6 addresses.

d. In the Choose the Cluster's Solution field, select the applicable option and click Next:
n Check Point ClusterXL and then select High Availability or Load Sharing
n Gaia VRRP

4 In the Cluster member's properties window perform these steps for each Cluster Member
and click Next.
We assume you create a new cluster object from the scratch.

a. Click Add > New Cluster Member to configure each Cluster Member.
b. In the Cluster Name field, enter unique name for the Cluster Member object.
c. In the Cluster IPv4 Address , enter the unique Cluster Virtual IPv4 addresses for this
Cluster Member.
This is the main IPv4 address of the Cluster Member object.
d. In the Cluster IPv6 Address , enter the unique Cluster Virtual IPv6 addresses for this
Cluster Member.
This is the main IPv6 address of the Cluster Member object.

Important - You must define a corresponding IPv4 address for every IPv6
address. This release does not support pure IPv6 addresses.

e. In the Activation Key and Confirm Activation Key fields, enter a one-time password
that you entered in First Time Configuration Wizard during the installation of this
Cluster Member.
f. Click Initialize.
Management Server will try to establish SIC with each Cluster Member.
The Trust State field should show Trust established.
g. Click OK.

ClusterXL R81 Administration Guide      |      88

 Configuring the Cluster Object and Members

Step Instructions

5 In the Cluster Topology window, define a network type (network role) for each cluster
interface and define the Cluster Virtual IP addresses. Click Next.
The wizard automatically calculates the subnet for each cluster network and assigns it to the
applicable interface on each Cluster Member. The calculated subnet shows in the upper
section of the window.
The available network objectives are:
Network
Description
Objective

Cluster A cluster interface that connects to an internal or external network.

Interface Enter the Cluster Virtual IP addresses for each network (internal or
external).
In addition, see "Cluster IP Addresses on Different Subnets" on
page 148.

Cluster Sync A cluster Synchronization interface.

Interface In Load Sharing mode, you must define a synchronization interface.
n For Check Point Appliances or Open Servers:
The Synchronization Network is supported only on the lowest
VLAN tag of a VLAN interface.
n For Small Office appliances (models lower than 3000 series):
You can only select 1st sync and only for the LAN2/SYNC
interface.
You cannot configure VLANs on the Synchronization interface.

Important - Make sure that you do not define IPv6 address

for Sync interfaces. The wizard does not let you define an
interface with an IPv6 address as a Sync interface.

Private An interface that is not part of the cluster.

Cluster does not monitor this interface.
Cluster failover does not occur if a fault occurs on this interface.
This option is recommended for the dedicated management
interface.

6 In the Cluster Definition Wizard Complete window, click Finish.

After you complete the wizard, we recommend that you open the cluster object and complete the
configuration:
n Define Anti-Spoofing properties for each interface
n Change the Topology settings for each interface, if necessary
n Define the Network Type
n Configure other Software Blades, features and properties as necessary

ClusterXL R81 Administration Guide      |      89

 Configuring the Cluster Object and Members

Wizard for Small Office Appliances

The Small Office Cluster wizard is recommended for Centrally Managed Check Point appliances -
models lower than 3000 series.

To create a new Small Office cluster using Wizard Mode:

Step Instructions

1 In SmartConsole, click Objects menu > More object types > Network Object > Gateways
and Servers > Cluster > New Small Office Cluster.

2 In Check Point Security Gateway Cluster Creation window, click Wizard Mode.

3 In the Cluster General Properties window:

a. Enter a unique name for the cluster object.
b. Select the correct hardware type.
c. Click Next.

4 In the Cluster Members window:

a. Enter the member name and IPv4 addresses for each Cluster Member.
b. Enter the one-time password for SIC trust.
c. Click Next.
d. Management Server will try to establish SIC with the Primary Cluster Member.

5 In the Configure WAN Interface page, configure the Cluster Virtual IPv4 address.

6 Define the Cluster Virtual IPv4 addresses for the other cluster interfaces.

7 Click Next, and then Finish to complete the wizard.

After you complete the wizard, we recommend that you open the cluster object and complete the
configuration:
n Define Anti-Spoofing properties for each interface
n Change the Topology settings for each interface, if necessary
n Define the Network Type
n Configure other Software Blades, features and properties as necessary

ClusterXL R81 Administration Guide      |      90

 Configuring the Cluster Object and Members

Using the Classic Mode in SmartConsole

Step Description

1 Connect with SmartConsole to the Security Management Server or Domain Management

Server that should manage this ClusterXL.

2 From the left navigation panel, click Gateways & Servers .

3 Create a new Cluster object in one of these ways:

n From the top toolbar, click the New ( ) > Cluster > Cluster.
n In the top left corner, click Objects menu > More object types > Network Object >
Gateways and Servers > Cluster > New Cluster.
n In the top right corner, click Objects Pane > New > More > Network Object >
Gateways and Servers > Cluster > Cluster.

4 In the Check Point Security Gateway Creation window, click Classic Mode.
The Gateway Cluster Properties window opens.

5 On the General Properties page > Machine section:

a. In the Name field, make sure you see the configured applicable name for this ClusterXL
object.
b. In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the Cluster
Member's First Time Configuration Wizard.
Make sure the Security Management Server or Multi-Domain Server can connect to
these IP addresses.

6 On the General Properties page > Platform section, select the correct options:

a. In the Hardware field:

If you install the Cluster Members on Check Point Appliances, select the correct
appliances series.
If you install the Cluster Members on Open Servers, select Open server.
b. In the Version field, select R81.
c. In the OS field, select Gaia.

7 On the General Properties page:

a. On the Network Security tab, make sure the ClusterXL Software Blade is selected.
b. Enable the additional applicable Software Blades on the Network Security tab and on
the Threat Prevention tab.

ClusterXL R81 Administration Guide      |      91

 Configuring the Cluster Object and Members

Step Description

8 On the Cluster Members page:

a. Click Add > New Cluster Member.

The Cluster Member Properties window opens.
b. In the Name field, enter the applicable name for this Cluster Member object.
c. Configure the main physical IP address(es) for this Cluster Member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the Cluster
Member's First Time Configuration Wizard.
Make sure the Security Management Server or Multi-Domain Server can connect to
these IP addresses.
Note - You can configure the Cluster Virtual IP address to be on a different
network than the physical IP addresses of the Cluster Members. In this case,
you must configure the required static routes on the Cluster Members.
d. Click Communication.
e. In the One-time password and Confirm one-time password fields, enter the same
Activation Key you entered during the Cluster Member's First Time Configuration Wizard.
f. Click Initialize.
g. Click Close.
h. Click OK.
i. Repeat Steps a-h to add the second Cluster Member, and so on.

If the Trust State field does not show Trust established, perform these steps:
a. Connect to the command line on the Cluster Member.
b. Make sure there is a physical connectivity between the Cluster Member and the
Management Server (for example, pings can pass).
c. Run:
cpconfig
d. Enter the number of this option:
Secure Internal Communication
e. Follow the instructions on the screen to change the Activation Key.
f. In SmartConsole, click Reset.
g. Enter the same Activation Key you entered in the cpconfig menu.
h. In SmartConsole, click Initialize.

9 On the ClusterXL and VRRP page:

a. In the Select the cluster mode and configuration section, select the applicable mode:
n High Availability and ClusterXL
n Load Sharing and Multicast or Unicast
b. In the Tracking section, select the applicable option.
c. In the Advanced Settings section:

ClusterXL R81 Administration Guide      |      92

 Configuring the Cluster Object and Members

Step Description

l If you selected the High Availability mode, then:

i. Optional: Select Use State Synchronization.

Best Practice - We recommend to select this option.

For more information, click the (?) button in the top right corner.
ii. Optional: Select Use Virtual MAC.
For more information, see sk50840.
iii. Select the Cluster Member recovery method.
For more information, click the (?) button in the top right corner.

l If you selected the Load Sharing > Multicast mode, then:

i. Optional: Select Use Sticky Decision Function.
This option is available only for clusters R80.10 and lower.
For more information, click the (?) button in the top right corner.
ii. Select the connection sharing method between the Cluster Members.
For more information, click the (?) button in the top right corner.

l If you selected the Load Sharing > Unicast mode, then:

i. Optional: Select Use Sticky Decision Function.
This option is available only for clusters R80.10 and lower.
For more information, click the (?) button in the top right corner.
ii. Optional: Select Use Virtual MAC.
For more information, see sk50840.
iii. Select the connection sharing method between the Cluster Members.
For more information, click the (?) button in the top right corner.

ClusterXL R81 Administration Guide      |      93

 Configuring the Cluster Object and Members

Step Description

10 On the Network Management page:

a. Select each interface and click Edit. The Network: <Name of Interface> window opens.
b. From the left tree, click the General page.
c. In the General section, in the Network Type field, select the applicable type:
n For cluster traffic interfaces, select Cluster.
Make sure the Cluster Virtual IPv4 address and its Net Mask are correct.
n For cluster synchronization interfaces, select Sync or Cluster+Sync .

Notes:
l We do not recommend the configuration Cluster+Sync .

l Check Point cluster supports only these settings:

o One Sync interface.

o One Cluster+Sync interface.
o One Sync interface and one Cluster+Sync interface.

l For Check Point Appliances or Open Servers:

The Synchronization Network is supported only on the lowest

VLAN tag of a VLAN interface.
n For interfaces that do not pass the traffic between the connected networks, select
Private.
d. In the Member IPs section, make sure the IPv4 address and its Net Mask are correct on
each Cluster Member.
Notes:
n For a ClusterXL in High Availability mode that is deployed in a Cloud
environment (Geo Cluster):
You can configure IP addresses that belong to different networks on
cluster synchronization interfaces and on cluster traffic interfaces.
n For cluster traffic interfaces, you can configure the Cluster Virtual IP
address to be on a different network than the physical IP addresses of
the Cluster Members. In this case, you must configure the required static
routes on the Cluster Members. See "Configuring Cluster Addresses on
Different Subnets" on page 149.
e. In the Topology section:
n Make sure the settings are correct in the Leads To and Security Zone fields.
n Make sure to enable the Anti-Spoofing.

11 Click OK.

12 Publish the SmartConsole session.

ClusterXL R81 Administration Guide      |      94

 Configuring the Cluster Object and Members

Changing the Settings of Cluster Object in SmartConsole

The Cluster Gateway Properties window in a cluster object contains many different ClusterXL properties,
as well as other properties related to Security Gateway and Software Blades functionality.
This section includes only the properties and procedures directly related to ClusterXL.

Configuring General Properties

1. In the Name field, enter a unique name for this cluster object.
2. In the IPv4 Address field, enter the unique Cluster Virtual IPv4 addresses for this cluster. This is the
main IPv4 address of the cluster object.
3. In the Cluster IPv6 Address field, enter the unique Cluster Virtual IPv6 addresses for this cluster.
This is the main IPv6 address of the cluster object.

Important - You must define a corresponding IPv4 address for every IPv6
address. This release does not support pure IPv6 addresses.

4. In the Hardware field, select the correct hardware platform.

5. In the Version field, select the correct Check Point version.
6. In the OS field, select the correct operating system.
7. Configure the applicable cluster type:
n To work with ClusterXL or with VRRP on Gaia, select ClusterXL.
Go to the ClusterXL and VRRP pane and configure the applicable settings.
n To work with any other cluster mode, clear ClusterXL.
Go to the 3rd Party Configuration pane and configure the applicable settings.
8. On the Network Security tab, enable other Software Blades as necessary.
9. Click OK.
10. Publish the SmartConsole session.

Working with Cluster Topology

IPv6 Considerations
To activate IPv6 functionality for an interface, define an IPv6 address for the applicable interface on each
Cluster Member and in the cluster object. All interfaces configured with an IPv6 address must also have a
corresponding IPv4 address. If an interface does not require IPv6, only the IPv4 definition address is
necessary.

Note - You must configure synchronization interfaces with IPv4 addresses only. This is
because the synchronization mechanism works using IPv4 only. All IPv6 information
and states are synchronized using this interface.

ClusterXL R81 Administration Guide      |      95

 Configuring the Cluster Object and Members

1. Connect with SmartConsole to the Security Management Server or Domain Management Server
that manages this cluster.
2. From the left navigation panel, click Gateways & Servers .
3. Open the cluster object.
4. From the left tree, click the Network Management page.
5. Select a cluster interface and click Edit.
6. From the left navigation tree, click General page:
a. In the General section, configure these settings for Cluster Virtual Interface:
n Network Type - one of these: Cluster, Sync , Cluster + Sync , Private
The available network types (network objectives) are:

Network
Description
Type

Cluster An interface that connects to an internal or external network.

Cluster A cluster interface that also works as a Synchronization interface.

+ Sync We do not recommend this configuration because it adds the Delta Sync
traffic to the interface.

Sync An interface used exclusively for cluster state synchronization.

Private An interface that is not part of the cluster. ClusterXL does not monitor the
state of this interface. As a result, there is no cluster failover if a fault
occurs with this interface. This option is recommended for the
management interface.

n Virtual IPv4 - Virtual IPv4 address assigned to this Cluster Virtual Interface
n Virtual IPv6 - Virtual IPv6 address assigned to this Cluster Virtual Interface

Important - You must define a corresponding IPv4 address for every IPv6
address. This release does not support the configuration of only IPv6
addresses.

7. In the Member IPs section, click Modify and configure these settings:
n Physical IPv4 address and Mask Length assigned to the applicable physical interface on each
Cluster Member
n Physical IPv6 address and Mask Length assigned to the applicable physical interface on each
Cluster Member

Important - You must define a corresponding IPv4 address for every IPv6
address. This release does not support the configuration of only IPv6
addresses.

In addition, see "Configuring Cluster Addresses on Different Subnets" on page 149.

8. In the Topology section, click Modify and configure these settings:

ClusterXL R81 Administration Guide      |      96

 Configuring the Cluster Object and Members

n Leads To - one of these: Internet (External), This Network (Internal)

n Security Zone - one of these: User defined, According to topology (ExternalZone,
InternalZone)
n Anti-Spoofing - whether to perform the Anti-Spoofing, and how to do it (Detect, Prevent)
9. From the left navigation tree, click QoS page:
a. In the Bandwidth section, configure these settings:
n Inbound Active - rate limit for inbound traffic
n Outbound Active - rate limit for outbound traffic
b. In the DiffServ and Low Latency classes section, configure the applicable classes.
10. From the left navigation tree, click Advanced page:
a. In the Multicast Restrictions section, configure the applicable settings for dropping multicast
packets
b. In the Interfaces Names section, configure the names of applicable interfaces
11. Click OK.
12. Publish the SmartConsole session.
13. Install the Access Control Policy on this cluster object.

Changing the Synchronization Interface

Important - Schedule a maintenance window, because changing the synchronization

interface can impact the traffic.

To change the IPv4 address on the synchronization interface on Cluster Members:

1. On each Cluster Member, change the IPv4 address on the Sync interface.
Use Gaia Portal, or Gaia Clish.
See the R81 Gaia Administration Guide.
2. Connect with SmartConsole to the Security Management Server or Domain Management Server
that manages this cluster.
3. From the left navigation panel, click Gateways & Servers .
4. Open the cluster object.
5. In the Gateway Cluster Properties window, click Network Management page.
6. Click Get Interfaces > Get Interfaces With Topology .
7. Make sure the settings are correct.
8. Select the Sync interface and click Edit.
9. From the left navigation tree, click General page.
10. In the General section, in the Network Type field, select Sync .

ClusterXL R81 Administration Guide      |      97

 Configuring the Cluster Object and Members

11. Click OK.

12. Publish the SmartConsole session.
13. Install the Access Control Policy on this cluster object.

To change the synchronization interface on Cluster Members to a new interface:

1. On each Cluster Member:
a. Configure a new interface that you will use as a new Sync interface.
b. Delete the IPv4 address from the old Sync interface.
Use Gaia Portal, or Gaia Clish.
See the R81 Gaia Administration Guide.
2. Connect with SmartConsole to the Security Management Server or Domain Management Server
that manages this cluster.
3. From the left navigation panel, click Gateways & Servers .
4. Open the cluster object.
5. In the Gateway Cluster Properties window, click Network Management page.
6. Click Get Interfaces > Get Interfaces With Topology .
7. Make sure the settings are correct.
8. Right-click on the old Sync interface and click Delete Interface.
9. Select the new interface and click Edit.
10. From the left navigation tree, click General page.
11. In the General section, in the Network Type field, select Sync .
12. Click OK.
13. In SmartConsole, install the Access Control Policy on this cluster object.
14. Publish the SmartConsole session.
15. Install the Access Control Policy on this cluster object.

Adding Another Member to an Existing Cluster

See "Adding Another Member to an Existing Cluster" on page 154.

Removing a Member from an Existing Cluster

See "Removing a Member from an Existing Cluster" on page 160.

ClusterXL R81 Administration Guide      |      98

 Configuring a ClusterXL in Bridge Mode

Configuring a ClusterXL in Bridge Mode

See the R81 Installation and Upgrade Guide - Chapter Special Scenarios for Security Gateways - Section
Deploying a Security Gateway or a ClusterXL in Bridge Mode.

ClusterXL R81 Administration Guide      |      99

 Advanced Features and Procedures

Advanced Features and Procedures

This section describes advanced configuration in cluster.

ClusterXL R81 Administration Guide      |      100

 Working with VPN in Cluster

Working with VPN in Cluster

This section describes the configuration of VPN in cluster.

Configuring VPN in Clusters

Configuring a cluster using SmartConsole is very similar to configuring a single Security Gateway.
All attributes of the VPN are defined in the Cluster object, except for two attributes that are defined for each
Cluster Member object.
1. In SmartConsole, open the cluster object.
2. In the left navigation tree, go to Cluster Members page.
3. Select each Cluster Member and click Edit.
The Cluster Member Properties window opens.
4. Go the VPN tab:
n In the Office Mode for Remote access section:

If you wish to use Office Mode for Remote Access, select Offer Manual Office Mode and
define the IP pool allocated to each Cluster Member.
n In the Certificate List with keys stored on the Security Gateway section:
If your Cluster Member supports hardware storage for IKE certificates, define the certificate
properties.
In that case, Management Server directs the Cluster Member to create the keys and supply
only the required material for creation of the certificate request.
The certificate is downloaded to the Cluster Member during policy installation.
5. Click OK to close the Cluster Member Properties window.
6. In the left navigation tree, go to ClusterXL and VRRP page.
7. Make sure to select Use State Synchronization.
This is required to synchronize IKE keys.
8. In the left navigation tree, go to Network Management > VPN Domain page.
9. Define the encryption domain of the cluster.
Select one of the two possible settings:
n All IP addresses behind Cluster Members based on Topology information. This is the
default option.
n Manually defined. Use this option if the cluster IP address is not on the member network, in
other words, if the cluster virtual IP address is on a different subnet than the Cluster Member
interfaces. In that case, select a network or group of networks, which must include the virtual
IP address of the cluster, and the network or group of networks behind the cluster.
10. Click OK to close the Gateway Cluster Properties window.
11. Install the Access Control Policy on the cluster.

ClusterXL R81 Administration Guide      |      101

 Working with VPN in Cluster

Defining VPN Peer Clusters with Separate Management

Servers
When working with a VPN peer that is a Check Point Cluster, and the VPN peer is managed by a different
Management Server, do NOT define another cluster object. Instead, do the following:
1. In SmartConsole, go to Objects menu > More object types > Network Object > Gateways and
Servers > More > New Externally Managed VPN Gateway .
The Externally Managed Check Point Gateway window opens.
2. In the General Properties page, configure the name and the IP address.
3. In the Topology page, click New to add the external and internal cluster interfaces on the VPN peer.
4. In the VPN Domain section of the Topology page, define the encryption domain of the externally
managed Security Gateway to be behind the internal Virtual IP address of the Security Gateway.
If the encryption domain is just one subnet, select All IP addresses behind Gateway based on
Topology information.
If the encryption domain includes more than one subnet, select Manually defined.

5. Click OK.
6. Install the Access Control Policy on the cluster.

ClusterXL R81 Administration Guide      |      102

 Working with NAT in Cluster

Working with NAT in Cluster

This section describes the configuration of NAT in cluster.

Cluster Fold and Cluster Hide

Network Address Translation (NAT) is a fundamental aspect of the way ClusterXL works.
n When a Cluster Member establishes an outgoing connection towards the Internet, the source
address in the outgoing packets, is the physical IP address of the Cluster Member interface.
The source IP address is changed using NAT to that of the external Virtual IP address of the cluster.
This address translation is called "Cluster Hide".
l When working with VRRP on Gaia cluster, this corresponds to the default setting in the
ClusterXL and VRRP page of the cluster object of Hide Cluster Members outgoing traffic
behind the Cluster IP address being selected.
l When working with VRRP on IPSO cluster, this corresponds to the default setting in the 3rd
Party Configuration page of the cluster object of Hide Cluster Members' outgoing traffic
behind the Cluster's IP address being selected.
n When a client establishes an incoming connection to external (virtual) address of the cluster,
ClusterXL changes the destination IP address using NAT to that of the physical external address of
one of the Cluster Members. This address translation is called "Cluster Fold".
l When working with VRRP on Gaia cluster, this corresponds to the default setting in the
ClusterXL and VRRP page of the cluster object of Forward Cluster incoming traffic to
Cluster Members IP address being selected.
l When working with IPSO IP Clustering cluster, this corresponds to the default setting in the
3rd Party Configuration page of the cluster object of Forward Cluster incoming traffic to
Cluster Members' IP addresses being selected.

Configuring NAT in Cluster

Network Address Translation (NAT) can be performed on a Cluster, in the same way as it is performed on
a Security Gateway.
This NAT is in addition to the automatic "Cluster Fold" and "Cluster Hide" address translations.
To configure NAT, edit the Cluster object, and in the Cluster Properties window, click the NAT page. Do
NOT configure the NAT tab of the Cluster Member object.

Configuring NAT on a Cluster Member

It is possible to perform Network Address Translation (NAT) on a non-cluster interface of a Cluster
Member.
A possible scenario for this is if the non-Cluster interface of the Cluster Member is connected to another
(non-cluster) internal Security Gateway, and you wish to hide the address of the non-Cluster interface of
the Cluster Member.

ClusterXL R81 Administration Guide      |      103

 Working with NAT in Cluster

Performing this NAT means that when a packet originates behind or on the non-Cluster interface of the
Cluster Member, and is sent to a host on the other side of the internal Security Gateway, the source
address of the packet will be translated.

To configure NAT on a non-cluster interface of a Cluster Member:

1. Edit the Cluster object.
2. In the Cluster Member page, edit the Cluster Member object.
3. In the Cluster Member Properties window, click the NAT tab.
4. Configure Static or Hide NAT as applicable.
5. Install the Access Control Policy on the cluster.

ClusterXL R81 Administration Guide      |      104

 Working with VLANs in Cluster

Working with VLANs in Cluster

A VLAN switch tags packets that originate in a VLAN with a four-byte header that specifies, which switch
port it came from
No packet is allowed to go from a switch port in one VLAN to a switch port in another VLAN, apart from
ports ("global" ports) that are defined so that they belong to all the VLANs.
The Cluster Member is connected to the global port of the VLAN switch, and this logically divides a single
physical port into many VLAN ports each associated with a VLAN tagged interface (VLAN interface) on the
Cluster Member.
When defining VLAN tags on an interface, cluster IP addresses can be defined only on the VLAN interfaces
(the tagged interfaces).
Defining a cluster IP address on a physical interface that has VLANs is not supported.
This physical interface has to be defined with the Network Type Private.
ClusterXL (including VSX) supports the Synchronization Network (CCP packets that carry Delta Sync
information) only on the lowest VLAN ID (VLAN tag).
For example, if three VLANs with IDs 10, 20 and 30 are configured on interface eth1, then you can use
only the VLAN interface eth1.10 for the State Synchronization.
This is the default interface monitoring in Check Point cluster:

Interface Monitoring in
Monitoring in VSX Cluster
type ClusterXL (non-VSX)

Physical Monitors all cluster Monitors all cluster interfaces.

interfaces interfaces.

VLAN Monitors only lowest VSX High Availability (non-VSLS):

interfaces VLAN ID configured
on a physical n Monitors only lowest and highest VLAN IDs configured on
interface. a physical interface.
n Monitors only lowest VLAN ID, if both VLAN IDs reside on
the same Virtual System.

Monitors only lowest Virtual System Load Sharing:

and highest VLAN IDs
configured on a n Monitors all VLAN IDs configured on a physical interface
physical interface. on each Virtual System.
n When a Virtual System is connected to a Virtual Switch with
the same physical interface and a lower VLAN ID, the wrp
interface that leads to the Virtual Switch is considered the
lowest VLAN ID for the physical interface.

You can customize the default monitoring of VLAN IDs:

ClusterXL R81 Administration Guide      |      105

 Working with VLANs in Cluster

Need to Monitoring in ClusterXL (non-

Monitoring in VSX Cluster
monitor VLAN VSX)

Only the Enabled by default. Must disable the monitoring of all VLAN IDs - set
lowest VLAN the value of the kernel parameter fwha_
ID monitor_all_vlan to 0.
See sk92826.

Only the Enabled by default. VSX High Availability (non-VSLS): Enabled by

lowest and Controlled by the kernel default.
highest VLAN parameter fwha_monitor_ Controlled by the kernel parameter fwha_
IDs low_high_vlans. monitor_low_high_vlans.
See sk92826. See sk92826.

All VLAN IDs Disabled by default. Virtual System Load Sharing: Disabled by default.
Controlled by the kernel Controlled by the kernel parameter fwha_
parameter fwha_monitor_ monitor_all_vlan.
all_vlan. See sk92826.
See sk92826.

Only specific Disabled by default. Disabled by default.

VLAN IDs Controlled by the kernel Controlled by the kernel parameter fwha_
parameter fwha_monitor_ monitor_specific_vlan.
specific_vlan. See sk92784.
See sk92784.

ClusterXL R81 Administration Guide      |      106

 Configuring Link Monitoring on the Cluster Interfaces

Configuring Link Monitoring on the Cluster

Interfaces

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
This procedure lets you configure the Cluster Member to monitor only the physical link on the cluster
interfaces (instead of monitoring the Cluster Control Protocol (CCP) packets):
n If a link disappears on the configured interface, the Cluster Member changes the interface's state to
DOWN.
This causes the Cluster Member to change its state to DOWN.
n If a link appears again on the configured interface, the Cluster Member changes the interface's state
back to UP.
This causes the Cluster Member to change its state back to ACTIVE or STANDBY.
See "Viewing Cluster State" on page 203.

ClusterXL R81 Administration Guide      |      107

 Configuring Link Monitoring on the Cluster Interfaces

Procedure

Step Instructions

1 Connect to the command line on the Cluster Member.

2 Log in to the Expert mode.

3 See if the $FWDIR/conf/cpha_link_monitoring.conf file already exists:

stat $FWDIR/conf/cpha_link_monitoring.conf

4 If the $FWDIR/conf/cpha_link_monitoring.conf file already exists, create a backup

copy:
cp -v $FWDIR/conf/cpha_link_monitoring.conf{,_BKP}
If the $FWDIR/conf/cpha_link_monitoring.conf file does not exist, create it:
touch $FWDIR/conf/cpha_link_monitoring.conf

5 Edit the $FWDIR/conf/cpha_link_monitoring.conf file:

vi $FWDIR/conf/cpha_link_monitoring.conf

6 n To monitor the link only on specific interfaces:

Enter the names of the applicable interfaces - each name on a new separate line.
Example:
eth2
eth4

n To monitor the link on all interfaces:

Enter only this word:
all

7 Save the changes in the file and exit the editor.

ClusterXL R81 Administration Guide      |      108

 Configuring Link Monitoring on the Cluster Interfaces

Step Instructions

8 Reboot the Cluster Member.

Important - This can cause a failover.

Best Practices:

n In High Availability cluster

1. Perform the configuration steps on all Cluster Members
2. Reboot all the Standby Cluster Members
3. Initiate a manual failover on the Active Cluster Member
4. Reboot the former Active Cluster Member
n In Load Sharing Unicast cluster
1. Perform the configuration steps on all Cluster Members
2. Reboot all the non-Pivot Cluster Members
3. Initiate a manual failover on the Pivot Cluster Member
4. Reboot the former Pivot Cluster Member
n In Load Sharing Multicast cluster
1. Perform the configuration steps on all Cluster Members
2. Reboot all Cluster Members except one
3. Initiate a manual failover on the remaining Cluster Member
4. Reboot the remaining Cluster Member

Note - See "Initiating Manual Cluster Failover" on page 189.

ClusterXL R81 Administration Guide      |      109

 Working with Bond Interfaces in Cluster

Working with Bond Interfaces in Cluster

This section describes the configuration of Bond Interfaces and Group of Bonds in cluster.

Bonding (Link Aggregation) Terminology

n Link Aggregation (Interface Bonding): Networking technology that binds multiple physical
interfaces together into one virtual interface.
n Bond: A group of physical interfaces that operate together as one virtual interface and share an IP
address and MAC address. A bond is identified by the cluster by its Bond ID (for example: bond0).
n Bond Interface: The logical representation of the bond.
n Slave (enslaved interface): A physical interface that is a member of a bond. Slaves do not have an
IP Address and in some cases share the same MAC address.

ClusterXL R81 Administration Guide      |      110

 Bond Interfaces (Link Aggregation)

Bond Interfaces (Link Aggregation)

Check Point security devices support Link Aggregation, a technology that joins multiple physical interfaces
into one virtual interface, known as a bond interface.
The bond interface share the load among many interfaces, which gives fault tolerance and increases
throughput. Check Point devices support the IEEE 802.3ad Link Aggregation Control Protocol (LCAP) for
dynamic link aggregation.

Item Description

1 Security Gateway

1A Interface 1

1B Interface 2

2 Bond Interface

3 Router

A bond interface (also known as a bonding group or bond) is identified by its Bond ID (for example:
bond1) and is assigned an IP address. The physical interfaces included in the bond are called slaves and
do not have IP addresses.

ClusterXL R81 Administration Guide      |      111

 Bond Interfaces (Link Aggregation)

You can configure a bond interface to use one of these functional strategies:
n High Availability (Active/Backup): Gives redundancy when there is an interface or a link failure.
This strategy also supports switch redundancy. Bond High Availability works in Active/Backup
mode - interface Active/Standby mode. When an Active slave interface is down, the connection
automatically fails over to the primary slave interface. If the primary slave interface is not available,
the connection fails over to a different slave interface.
n Load Sharing (Active/Active): All slave interfaces in the UP state are used simultaneously. Traffic is
distributed among the slave interfaces to maximize throughput. Bond Load Sharing does not
support switch redundancy.

Note - Bonding Load Sharing mode requires SecureXL to be enabled on

Security Gateway or each Cluster Member.

You can configure Bond Load Sharing to use one of these modes:
l Round Robin - Selects the Active slave interfaces sequentially.
l 802.3ad (LACP) - Dynamically uses Active slave interfaces to share the traffic load. This
mode uses the LACP protocol, which fully monitors the interface link between the Check Point
Security Gateway and a switch.
l XOR - All slave interfaces in the UP state are Active for Load Sharing. Traffic is assigned to
Active slave interfaces based on the transmit hash policy: Layer 2 information (XOR of
hardware MAC addresses), or Layer 3+4 information (IP addresses and Ports).
For Bonding High Availability mode and for Bonding Load Sharing mode:
n The number of bond interfaces that can be defined is limited by the maximal number of interfaces
supported by each platform. See the R81 Release Notes.
n Up to 8 physical slave interfaces can be configured in a single bond interface.

ClusterXL R81 Administration Guide      |      112

 Bond High Availability Mode in Cluster

Bond High Availability Mode in Cluster

When dealing with mission-critical applications, an enterprise requires its network to be highly available.
Clustering provides redundancy at the Security Gateway level.
Without Bonding, redundancy of Network Interface Cards (NICs) or of the switches on either side of the
Security Gateway are only possible in a cluster, and only by failover from one Cluster Member to another
Cluster Member.

ClusterXL R81 Administration Guide      |      113

 Simple Redundant Topology

Simple Redundant Topology

You can have redundancy of clustering without Bonding.
If a switch or Cluster Member fails, a High Availability cluster solution provides system redundancy.
For example, you can have a redundant system with two synchronized Cluster Members deployed in a
simple redundant topology.

Item Description

1 Cluster Member GW1 with interfaces connected to the external switches (5 and 6)

2 Cluster Member GW2 with interfaces connected to the external switches (5 and 6)

3 Interconnecting network C1

4 Interconnecting network C2

5 Switch S1

6 Switch S2

If Cluster Member GW1 (1), its NIC, or switch S1 (5) fails, Cluster Member GW2 (2) becomes the only
Active member, connecting to switch S2 (6) over network C2 (4).
If any component fails (Cluster Member, NIC, or switch), the result of the failover is that no further
redundancy exists.
A further failure of any active component completely stops network traffic through this cluster.

ClusterXL R81 Administration Guide      |      114

 Fully Meshed Redundancy

Fully Meshed Redundancy

The Bonding High Availability mode, when deployed with ClusterXL, enables a higher level of reliability by
providing granular redundancy in the network. This granular redundancy is achieved by using a fully
meshed topology, which provides for independent backups for both NICs and switches.
A fully meshed topology further enhances the redundancy in the system by providing a backup to both the
interface and the switch, essentially backing up the cable. Each Cluster Member has two external
interfaces, one connected to each switch.

Item Description

1 Cluster Member GW1 with interfaces connected to the external switches (4 and 5)

2 Cluster Member GW2 with interfaces connected to the external switches (4 and 5)

3 Interconnecting network

4 Switch S1

5 Switch S2

In this scenario:
n GW1 and GW2 are Cluster Members in the High Availability mode, each connected to the two
external switches
n S1 and S2 are external switches
n Item 3 are the network connections
If any of the interfaces on a Cluster Member that connect to an external switch fails, the other interface
continues to provide the connectivity.
If any Cluster Member, its NIC, or switch fails, the other Cluster Member, connecting to switch S2 over
network C2. If any component fails (Cluster Member , NIC, or switch), the result of the failover is that no
further redundancy exists. A further failure of any active component completely stops network traffic.
Bonding provides High Availability of NICs. If one fails, the other can function in its place.

ClusterXL R81 Administration Guide      |      115

 Bond Failover in High Availability Mode

Bond Failover in High Availability Mode

In Bond High Availability mode, configured on Cluster Members, bond internal failover can occur in one of
these cases:
n An active interface detects a failure in the link state, in a monitored interface.
n ClusterXL detects a failure in sending or receiving Cluster Control Protocol (CCP) packets.
Either of these failures causes a failover within the interface bond, or between Cluster Members,
depending on the circumstances.
When a failure is detected, a log is recorded:
1. Connect with SmartConsole to the Management Server.
2. From the left navigation panel, click Logs & Monitor > Logs .

ClusterXL R81 Administration Guide      |      116

 Configuring a Bond Interface in High Availability Mode

Configuring a Bond Interface in High Availability Mode

On each Cluster Member, follow the instructions in the R81 Gaia Administration Guide - Chapter Network
Management - Section Network Interfaces - Section Bond Interfaces (Link Aggregation).

ClusterXL R81 Administration Guide      |      117

 Making Sure the Bond Interface is Functioning Properly

Making Sure the Bond Interface is Functioning Properly

Examine the bond information to make sure the bond interface is UP:

Step Instructions

1 Connect to the command line on each Cluster Member.

2 Examine the cluster interfaces in one of these ways:

n In Gaia Clish:
show cluster members interfaces
all
n In the Expert mode:
cphaprob -am if

See "Viewing Cluster Interfaces" on page 214.

3 Examine the bond interfaces in one of these ways:

n In Gaia Clish:
show cluster bond {all | name
<Name of Bond>}

show bonding groups

n In the Expert mode:
cphaprob show_bond <Name of
Bond>

See "Viewing Bond Interfaces" on page 218.

ClusterXL R81 Administration Guide      |      118

 Failover Support for VLANs

Failover Support for VLANs

In Bonding High Availability mode, ClusterXL monitors VLAN IDs for connectivity failure or
miscommunication, and initiate a failover when a failure is detected.
In a VLAN-enabled switched environment, ClusterXL monitors the VLAN with the lowest ID number. The
monitoring is conducted by sending ClusterXL Control Protocol (CCP) packets on round-trip paths at a set
interval.
The lowest VLAN ID indicates the status of the physical connection. This VLAN ID is always monitored, and
a connectivity failure causes ClusterXL to initiate a failover.
ClusterXL does not detect a VLAN configuration problems on switches.
For more information, see "Working with VLANs in Cluster" on page 105.

ClusterXL R81 Administration Guide      |      119

 Sync Redundancy

Sync Redundancy
The use of more than one physical synchronization interface (1st sync, 2nd sync, 3rd sync ) for
synchronization redundancy is not supported. For synchronization redundancy, you can use bond
interfaces.

Requirements and Limitations:

n The bond slave interfaces on each Cluster Member must connect to the same switch or VLAN (for
example, physical interface eth1 on all Cluster Members must connect to the same switch).
n We recommend that interfaces and other network hardware support the IEEE 802.3 bond mode.
n If you use a Bond in High Availability mode, you must add slave interfaces to the bonding group in
the same order on all Cluster Members.

Important - See "Supported Topologies for Synchronization Network" on page 41.

To configure bond interfaces for Sync High Availability:

1. Configure a bond interface on each Cluster Member with unused slave interfaces. See "Configuring
a Bond Interface in High Availability Mode" on page 125.
2. Connect with SmartConsole to the Management Server.
3. From the left navigation panel, click Gateways & Servers .
4. Open the cluster object.
5. From the left tree, click Network Management.
6. At the top, click Get Interfaces > Get Interfaces With Topology .
7. Select the applicable interface and click Edit.
8. In the General section, in the Network Type field, select Sync .
9. Click OK.
10. Install the Access Control Policy on this cluster object.
11. On each Cluster Member, make sure that the Sync interfaces are in the bond.
Examine the cluster interfaces in one of these ways:
n In Gaia Clish:

show cluster members interfaces all

n In the Expert mode:

cphaprob -am if

See "Viewing Cluster Interfaces" on page 214.

ClusterXL R81 Administration Guide      |      120

 Configuring Bond High Availability in VRRP Cluster

Configuring Bond High Availability in VRRP Cluster

The R80.20 version introduced an improved Active/Backup Bond mechanism (Enhanced Bond) when
working in ClusterXL.
If you work with ClusterXL, the Enhanced Bond feature is enabled by default, and no additional
configuration is required.
If you change your cluster configuration from ClusterXL to VRRP (MCVR & VRRP), or configure the VRRP
(MCVR & VRRP) cluster from scratch, the Enhanced Bond feature is disabled by default.
If you change your cluster configuration from VRRP to ClusterXL, you must manually enable the Enhanced
Bond feature.
To enable the Enhanced Bond feature in VRRP Cluster, set the value of the kernel parameter fwha_
bond_enhanced_enable to 1 on each VRRP Cluster Member. You can set the value of the kernel
parameter temporarily, or permanently.

Setting the value of the kernel parameter temporarily

Important - This change does not survive reboot.

Step Description

1 Connect to the command line on each VRRP Cluster Member.

2 Log in to the Expert mode.

3 Set the value of the kernel parameter fwha_bond_enhanced_enable to 1:

fw ctl set int fwha_bond_enhanced_enable 1

4 Make sure the value of the kernel parameter fwha_bond_enhanced_enable was set to
1:
fw ctl get int fwha_bond_enhanced_enable

ClusterXL R81 Administration Guide      |      121

 Configuring Bond High Availability in VRRP Cluster

Setting the value of the kernel parameter permanently

Step Description

1 Connect to the command line on each Cluster Member.

2 Log in to the Expert mode.

3 Back up the current $FWDIR/boot/modules/fwkern.conf file:

cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}

4 Edit the current $FWDIR/boot/modules/fwkern.conf file:

vi $FWDIR/boot/modules/fwkern.conf

5 Add this line to the file (spaces and comments are not allowed):
fwha_bond_enhanced_enable=1

6 Save the changes in the file and exit the editor.

7 Reboot the Cluster Member.

8 Make sure the value of the kernel parameter fwha_bond_enhanced_enable was set to
1:
fw ctl get int fwha_bond_enhanced_enable

Important - If you change your cluster configuration from VRRP to ClusterXL, you must
remove the kernel parameter configuration from each Cluster Member.

ClusterXL R81 Administration Guide      |      122

 Bond Load Sharing Mode in Cluster

Bond Load Sharing Mode in Cluster

In Bond Load Sharing mode:
n All slave interfaces are active, and connections are balanced between the bond slave interfaces,
similar to the way ClusterXL Load Sharing balances connections between Cluster Members.
n Each connection is assigned to a specific slave interface. For the individual connection, only one
slave interface is active. On failure of that interface, the bond fails over the connection to one of the
other slave interfaces, which adds the failed interface connection to the connections it is already
handling.
n All the slave interfaces of a bond must be connected to the same switch. The switch itself must
support and be configured for Bonding, by the same standard (for example, 802.3ad, or XOR) as
the Security Gateway bond.

Important - Bond Load Sharing mode requires SecureXL to be enabled on each

Cluster Member (this is the default).

ClusterXL R81 Administration Guide      |      123

 Bond Failover in Load Sharing Mode

Bond Failover in Load Sharing Mode

In Bond Load Sharing mode, configured on Cluster Members, bond internal failover can occur in one of
these cases:
n An active interface detects a failure in the link state, in a monitored interface.
n ClusterXL detects a failure in sending or receiving Cluster Control Protocol (CCP) packets.
Either of these failures will induce a failover within the interface bond, or between Cluster Members,
depending on the circumstances.
When a failure is detected, a log is recorded:
1. Connect with SmartConsole to the Management Server.
2. From the left navigation panel, click Logs & Monitor > Logs .

ClusterXL R81 Administration Guide      |      124

 Configuring a Bond Interface in High Availability Mode

Configuring a Bond Interface in High Availability Mode

On each Cluster Member, follow the instructions in the R81 Gaia Administration Guide - Chapter Network
Management - Section Network Interfaces - Section Bond Interfaces (Link Aggregation).

ClusterXL R81 Administration Guide      |      125

 Configuring Minimal Number of Required Slave Interfaces

Configuring Minimal Number of Required Slave Interfaces

ClusterXL considers a bond in Load Sharing mode to be in the "down" state when fewer than a minimal
number of required slave interfaces stay in the "up" state.
By default, the minimal number of required slave interfaces, which must stay in the "up" state in a bond of n
slave interfaces is n-1.
If one more slave interface fails (when n-2 slave interfaces stay in the "up" state), ClusterXL considers the
bond interface to be in the "down" state, even if the bond contains more than two slave interfaces.
If a smaller number of slave interfaces can pass the expected traffic, you can configure explicitly the
minimal number of required slave interfaces.
Follow "Configuring the Minimal Number of Required Slave Interfaces for Bond Load Sharing" on
page 193.

ClusterXL R81 Administration Guide      |      126

 Making Sure the Bond Interface is Functioning Properly

Making Sure the Bond Interface is Functioning Properly

Examine the bond information to make sure the bond interface is UP:

Step Instructions

1 Connect to the command line on each Cluster Member.

2 Examine the cluster interfaces in one of these ways:

n In Gaia Clish:
show cluster members interfaces
all
n In the Expert mode:
cphaprob -am if

See "Viewing Cluster Interfaces" on page 214.

3 Examine the bond interfaces in one of these ways:

n In Gaia Clish:
show cluster bond {all | name
<Name of Bond>}

show bonding groups

n In the Expert mode:
cphaprob show_bond <Name of
Bond>

See "Viewing Bond Interfaces" on page 218.

ClusterXL R81 Administration Guide      |      127

 Group of Bonds

Group of Bonds
In This Section:

Introduction
Group of Bonds, which is a logical group of existing Bond interfaces, provides additional link redundancy.

Example topology with Group of Bonds

n There is one router - R

n There are two switches that connect to the router R: SW-1 and SW-2
n There are two Cluster Members GW-A (Active) and GW-B (Standby)
n There are two Bond interfaces on each Cluster Member: Bond-1 and Bond-2
n On the Cluster Member GW-A:
l Bond-1 interface connects to the switch SW-1
l Bond-2 interface connects to the switch SW-2
n On the Cluster Member GW-B:
l Bond-1 interface connects to the switch SW-2
l Bond-2 interface connects to the switch SW-1

ClusterXL R81 Administration Guide      |      128

 Group of Bonds

Chain of events without Group of Bonds

1. The Cluster Member GW-A is the Active and the Cluster Member GW-B is the Standby.
2. On the Cluster Member GW-A, the Bond-1 interface fails.
3. On the Cluster Member GW-A, the Critical Device Interface Active Check reports its state as
"problem".

4. The Cluster Member GW-A changes its cluster state from Active to Down.
5. The cluster fails over - the Cluster Member GW-B changes its cluster state from Standby to
Active.
This is not the desired behavior, because the Cluster Member GW-A connects not only to the switch
SW-1, but also to the switch SW-2. In our example topology, there is no actual reason to fail-over from
the Cluster Member GW-A to the Cluster Member GW-B.
In order to overcome this problem, Cluster Members use the Group of Bonds consisting of Bond-1 and
Bond-2. The Group of Bonds fails only when both Bond interfaces fail on the Cluster Member. Only then
the cluster fails over.

ClusterXL R81 Administration Guide      |      129

 Group of Bonds

Chain of events with configured Group of Bonds

1. The Cluster Member GW-A is the Active and the Cluster Member GW-B is the Standby.
2. On the Cluster Member GW-A, the Bond-1 interface fails.
3. On the Cluster Member GW-A, the Critical Device Interface Active Check reports its state as
"problem".

4. The Cluster Member GW-A does not change its cluster state from Active to Down.
5. On the Cluster Member GW-A, the Bond-2 interface fails as well.
6. The Cluster Member GW-A changes its cluster state from Active to Down.
7. The cluster fails over - the Cluster Member GW-B changes its cluster state from Standby to
Active.

Creating a new Group of Bonds

This procedure lets you create a new Group of Bonds.

Procedure

Important - In a Cluster, you must configure all the Cluster Members in the same way.

1. Connect to the command line on the Cluster Member.

2. Log in to the Expert mode.
3. In VSX Cluster, switch to the context of the applicable Virtual System:

vsenv <VSID>

4. Modify the current $FWDIR/boot/modules/fwkern.conf file:

a. Backup the current file:

cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}

b. Edit the current file:

vi $FWDIR/boot/modules/fwkern.conf

ClusterXL R81 Administration Guide      |      130

 Group of Bonds

c. Add these two lines at the bottom of the file (spaces or comments are not allowed):

fwha_group_of_bonds_str=<Name for Group of Bonds>:<List of

all Bonds in this Group separated by comma>
fwha_arp_probe_method=1

Example:

fwha_group_of_bonds_str=GoB0:bond0,bond1;GoB1:bond2,bond3
fwha_arp_probe_method=1
Note - The kernel parameter "fwha_arp_probe_method"
configures the Cluster Member to use the Virtual IP address as the
Source IP address in the ARP Requests during the probing of the local
network.
d. Save the changes in the file and exit the editor.
5. Change the value of the kernel parameter fwha_group_of_bonds_str to add the Group of
Bonds on-the-fly:

fw ctl set str fwha_group_of_bonds_str '<Name for Group of

Bonds>:<List of all Bonds in this Group separated by comma>'

Example:

fw ctl set str fwha_group_of_bonds_str

'GoB0:bond0,bond1;GoB1:bond2,bond3'
Notes:
n The apostrophe characters are mandatory part of the syntax.
n Spaces are not allowed in the value of the kernel parameter fwha_
group_of_bonds_str.
6. Change the value of the kernel parameter fwha_arp_probe_method on-the-fly:

fw ctl set int fwha_arp_probe_method 1

7. Make sure the Cluster Member accepted the new configuration:

fw ctl get str fwha_group_of_bonds_str

fw ctl get int fwha_arp_probe_method

8. In SmartConsole, install the Access Control Policy on the cluster object.

Adding a Bond interface to the existing Group of Bonds

This procedure lets you add an additional bond interface to the existing Group of Bonds.

Procedure

Important - In a Cluster, you must configure all the Cluster Members in the same way.

ClusterXL R81 Administration Guide      |      131

 Group of Bonds

1. Connect to the command line on the Cluster Member.

2. In VSX Cluster, switch to the context of the applicable Virtual System:

vsenv <VSID>

3. Log in to the Expert mode.

4. Modify the current $FWDIR/boot/modules/fwkern.conf file:
a. Backup the current file:

cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}

b. Edit the current file:

vi $FWDIR/boot/modules/fwkern.conf

c. Edit the value of the kernel parameter fwha_group_of_bonds_str to add the Bond
interface to the existing Group of Bonds.
Example:

fwha_group_of_bonds_
str=GoB0:bond0,bond1;GoB1:bond2,bond3,bond4

d. Save the changes in the file and exit the editor.

5. Get the current value of the kernel parameter fwha_group_of_bonds_str and copy it:

fw ctl get str fwha_group_of_bonds_str

6. Reset the current value of the kernel parameter fwha_group_of_bonds_str:

fw ctl set str fwha_group_of_bonds_str ''

7. Make sure the Cluster Member reset the value of the kernel parameter fwha_group_of_
bonds_str:

fw ctl get str fwha_group_of_bonds_str

8. Change the value of the kernel parameter fwha_group_of_bonds_str to add the Bond
interface to the existing Group of Bonds on-the-fly:

fw ctl set str fwha_group_of_bonds_str '<Name for Group of

Bonds>:<List of all Bonds in this Group separated by comma>'

Example:

fw ctl set str fwha_group_of_bonds_str

'GoB0:bond0,bond1;GoB1:bond2,bond3,bond4'

ClusterXL R81 Administration Guide      |      132

 Group of Bonds

Notes:
n The apostrophe characters are mandatory part of the syntax.
n Spaces are not allowed in the value of the kernel parameter fwha_
group_of_bonds_str.
9. Make sure the Cluster Member accepted the new configuration:

fw ctl get str fwha_group_of_bonds_str

10. In SmartConsole, install the Access Control Policy on the cluster object.

Removing a Bond interface from the existing Group of Bonds

This procedure lets you remove a bond interface from an existing Group of Bonds.

Procedure

Important - In a Cluster, you must configure all the Cluster Members in the same way.

1. Connect to the command line on the Cluster Member.

2. Log in to the Expert mode.
3. In VSX Cluster, switch to the context of the applicable Virtual System:

vsenv <VSID>

4. Modify the current $FWDIR/boot/modules/fwkern.conf file:

a. Backup the current file:

cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}

b. Edit the current file:

vi $FWDIR/boot/modules/fwkern.conf

c. Edit the value of the kernel parameter fwha_group_of_bonds_str to remove the

Bond interface from the existing Group of Bonds.
Example:

fwha_group_of_bonds_str=GoB0:bond0,bond1;GoB1:bond2,bond3

d. Save the changes in the file and exit the editor.

5. Get the current value of the kernel parameter fwha_group_of_bonds_str and copy it:

fw ctl get str fwha_group_of_bonds_str

6. Reset the current value of the kernel parameter fwha_group_of_bonds_str:

fw ctl set str fwha_group_of_bonds_str ''

ClusterXL R81 Administration Guide      |      133

 Group of Bonds

7. Make sure the Cluster Member reset the value of the kernel parameter fwha_group_of_
bonds_str:

fw ctl get str fwha_group_of_bonds_str

8. Change the value of the kernel parameter fwha_group_of_bonds_str to remove the Bond
interface from the existing Group of Bonds on-the-fly:

fw ctl set str fwha_group_of_bonds_str '<Name for Group of

Bonds>:<List of all Bonds in this Group separated by comma>'

Example:

fw ctl set str fwha_group_of_bonds_str

'GoB0:bond0,bond1;GoB1:bond2,bond3
Notes:
n The apostrophe characters are mandatory part of the syntax.
n Spaces are not allowed in the value of the kernel parameter fwha_
group_of_bonds_str.
9. Make sure the Cluster Member accepted the new configuration:

fw ctl get str fwha_group_of_bonds_str

10. In SmartConsole, install the Access Control Policy on the cluster object.

Deleting a Group of Bonds

This procedure lets you delete an existing Group of Bonds.

Procedure

Important - In a Cluster, you must configure all the Cluster Members in the same way.

1. Connect to the command line on the Cluster Member.

2. Log in to the Expert mode.
3. In VSX Cluster, switch to the context of the applicable Virtual System:

vsenv <VSID>

4. Modify the current $FWDIR/boot/modules/fwkern.conf file:

a. Backup the current file:

cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}

b. Edit the current file:

vi $FWDIR/boot/modules/fwkern.conf

ClusterXL R81 Administration Guide      |      134

 Group of Bonds

c. Delete these two lines in the file:

fwha_group_of_bonds_str=<Name for Group of Bonds>:<List of

all Bonds in this Group separated by comma>
fwha_arp_probe_method=1

d. Save the changes in the file and exit the editor.

5. Reset the current value of the kernel parameter fwha_group_of_bonds_str:

fw ctl set str fwha_group_of_bonds_str ''

6. Make sure the Cluster Member reset the value of the kernel parameter fwha_group_of_
bonds_str:

fw ctl get str fwha_group_of_bonds_str

7. In SmartConsole, install the Access Control Policy on the cluster object.

Monitoring
To see the configured Groups of Bonds, run the "cphaprob show_bond_groups" command. See
"Viewing Bond Interfaces" on page 218.

Logs
Cluster Members generate some applicable logs.

Applicable log files

1. User Space logs:

n In non-VSX Cluster:
In the /var/log/messages files.
Output of the dmesg command.
n In VSX Cluster:
In the $FWDIR/log/fwk.elg file in the context of the applicable Virtual System.
2. Kernel Space logs:
n In the kernel module fw, enable the debug flags error and ioctl
n Kernel module cluster, enable the debug flag if

See the R81 Next Generation Security Gateway Guide - Chapter Kernel Debug on Security
Gateway.
The kernel debug shows:

ClusterXL R81 Administration Guide      |      135

 Group of Bonds

n A physical slave interface goes down/up.

n A Bond interface goes down/up (regardless of a change in the Cluster Member's state).
n A Group of Bonds goes down/up.

Note - These logs are generated only once per event.

Limitations
Specific limitations apply to a Group of Bonds.

List of Limitations

n The maximal length on the text string "<Name for Group of Bonds>" is 16 characters.
n The maximal length on this text string is 1024 characters:

<Name for Group of Bonds>:<List of all Bonds in this Group

separated by comma>

n You can configure the maximum of five Groups of Bonds on a Cluster Member or Virtual System.
n You can configure the maximum of five Bond interfaces in each Groups of Bonds.
n Group of Bonds feature does support Virtual Switches and Virtual Routers. Meaning, do not
configure Groups of Bonds in the context of these Virtual Devices.
n Group of Bonds feature supports only Bond interfaces that belong to the same Virtual System.
You cannot configure bonds that belong to different Virtual Systems into the same Group of
Bonds.
You must perform all configuration in the context of the applicable Virtual System.
n Group of Bonds feature does support Sync interfaces (an interface on a Cluster Member, whose
Network Type was set as Sync or Cluster+Sync in SmartConsole in cluster object).
n Group of Bonds feature does support Bridge interfaces.
n If a Bond interface goes down on one Cluster Member, the "cphaprob show_bond_groups"
command (see "Viewing Bond Interfaces" on page 218) on the peer Cluster Members also
shows the same Bond interface as DOWN.
This is because the peer Cluster Members stop receiving the CCP packets on that Bond interface
and cannot probe the local network to determine that their Bond interface is really working.
n After you add a Bond interface to the existing Group of Bonds, you must install the Access Control
Policy on the cluster object.
n After you remove a Bond interface from the existing Group of Bonds, you must install the Access
Control Policy on the cluster object.

ClusterXL R81 Administration Guide      |      136

 Performance Guidelines for Bond Interfaces

Performance Guidelines for Bond Interfaces

For optimal performance, follow these guidelines:
1. Configure static affinities of bond slave interfaces to CPU Cores.
2. Whenever possible, dedicate one processing core to each interface.
3. If there are more physical interfaces than CPU cores, then some CPU cores handle two or more
interfaces.
Use pairs of slave interface of the same position with internal and external bonds.
a. To view positions of slave interface in a bond, run in the Expert mode:

cat /proc/net/bonding/<Name of Bond Interface>

b. Note the sequence of the interfaces in the output.

Compare this sequence for the two bonds (external bond and its respective internal bond).
Slave interfaces that appear in the same position in the two bonds are interface pairs.
Set these pairs to be handled by one processing CPU core.

Example configuration

An appliance has:
n Four processing CPU cores:
core 0, core 1, core 2, and core 3
n Two bond interfaces:
bond0 with slave interfaces eth0, eth1, and eth2
bond1 with slave interfaces eth3, eth4, and eth5
In such case, two of the CPU cores need to handle two slave interfaces each.
An optimal configuration can be:

CPU core bond0 bond1

0 eth0 eth3

1 eth1 eth4

2 eth2

3 eth5

For more information, see the R81 Performance Tuning Administration Guide:
n Chapter SecureXL > Section SecureXL Commands and Debug > Section 'sim' and 'sim6' > Section
sim affinity.

ClusterXL R81 Administration Guide      |      137

 Performance Guidelines for Bond Interfaces

n Chapter CoreXL > Section Configuring Affinity Settings.

n Chapter CoreXL > Section Affinity Settings for 16000 and 26000 Appliances.

ClusterXL R81 Administration Guide      |      138

 Troubleshooting Issues with Bonded Interfaces

Troubleshooting Issues with Bonded Interfaces

Troubleshooting Workflow
1. View the logs from this cluster in SmartConsole > Logs & Monitor > Logs .
2. On the Cluster Members, examine the status of the bond interface in one of these ways:
n In Gaia Clish:

show cluster bond name <Name of Bond>

n In the Expert mode:

cphaprob show_bond <Name of Bond>

See "Viewing Bond Interfaces" on page 218.

3. If there is a problem, see if the physical link is down:
a. Look for a slave interface that reports the status of the link as "no".
b. Examine the cable connections and other hardware.
c. Examine the port configuration on the switch, to which this slave interface connects.
On a VSX Cluster Member, reboot is needed after these actions on a bond interface:
1. Changing a bond mode.
2. Adding a slave interface into an existing bond.

Note - Removing a slave interface from an existing bond, does not require a reboot.

Connectivity Delays on Switches

Connectivity delays may occur in switches during some internal bond failovers. With the various features
that are now included on some switches, it can take close to a minute for a switch to begin servicing a newly
connected interface.
These are suggestions for reducing the startup time after link failure.
1. Disable auto-negotiation on the relevant interface.
2. On Cisco switches, enable the PortFast feature (see the applicable Cisco documentation).
Warning - The PortFast feature should never be used on ports that connect to
switches or hubs. It is important that the Spanning Tree complete the
initialization procedure in these situations. Otherwise, these connections may
cause physical loops where packets are continuously forwarded (or even
multiply) in such a way that can cause the network to fail.
3. Disable STP on the switch ports (see the applicable switch vendor documentation).

ClusterXL R81 Administration Guide      |      139

 Advanced Cluster Configuration

Advanced Cluster Configuration

A number of synchronization and ClusterXL capabilities are controlled by kernel parameters.

Important: - In a Cluster, you must configure all the Cluster Members in the same way.

See the R81 Next Generation Security Gateway Guide - Chapter Working with Kernel Parameters on
Security Gateway.

ClusterXL R81 Administration Guide      |      140

 Controlling the Clustering and Synchronization Timers

Controlling the Clustering and Synchronization Timers

Best Practice - Do not change the default values.

These kernel parameters control the clustering and synchronization timers.

Parameter Description Default Value

fwha_timer_cpha_res The frequency of ClusterXL operations on the cluster. 1

Operations occur every number of milliseconds:
10 x (fwha_timer_cpha_res) x
(fwha_timer_base_res)

fwha_timer_sync_res The frequency of sync flush operations on the cluster. 1

Operations occur every number of milliseconds:
10 x (fwha_timer_sync_res) x
(fwha_timer_base_res)

fwha_timer_base_res Must be divisible by 10 with no remainders. 10

ClusterXL R81 Administration Guide      |      141

 Blocking New Connections Under Load

Blocking New Connections Under Load

Important - This section applies only to ClusterXL Load Sharing modes.

The reason for blocking new connections is that new connections are the main source of new Delta
Synchronization traffic. Delta Synchronization may be at risk, if new traffic continues to be processed at
high rate.
A related error message in cluster logs and in the /var/log/messages file is:

State synchronization is in risk

Reducing the amount of traffic passing through the Cluster Member protects the Delta Synchronization
mechanism. See sk43896: Blocking New Connections Under Load in ClusterXL.
These kernel parameters let you control how Cluster Member behave:

Kernel
Description
Parameter

fw_sync_ Controls how Cluster Member detect heavy loads and whether they start blocking new
block_ connections.
new_conns Load is considered heavy when the synchronization transmit queue of the Cluster
Member starts to fill beyond the value of the kernel parameter "fw_sync_buffer_
threshold".
n To enable blocking new connections under load, set the value of the "fw_sync_
block_new_conns" to 0.
n To disable blocking new connections under load, set the value of the "fw_sync_
block_new_conns" to -1 (must use the hex value 0xFFFFFFFF). This is the
default.

Note - Blocking new connections when sync is busy is only recommended for
ClusterXL Load Sharing deployments. While it is possible to block new
connections in ClusterXL High Availability mode, doing so does not solve
inconsistencies in sync, because the High Availability mode prevents that
from happening.

fw_sync_ Configures the maximum percentage of the buffer that may be filled before new
buffer_ connections are blocked (see the parameter "fw_sync_block_new_conns"
threshold above).
The default percentage value is 80, with a buffer size of 512.
By default, if more than 410 consecutive packets are sent without getting an ACK on
any one of them, new connections are dropped.

ClusterXL R81 Administration Guide      |      142

 Blocking New Connections Under Load

Kernel
Description
Parameter

fw_sync_ Determines the type of connections that can be opened while the system is in a blocking
allowed_ state.
protocols Thus, the user can have better control over the system behavior in cases of unusual
load.
The value of this kernel parameter is a combination of flags, each specifying a different
type of connection. The required value is the result of adding the separate values of
these flags.
Summary table:
Flag Value

ICMP_CONN_ALLOWED 1

TCP_CONN_ALLOWED 2 (except for data connections)

UDP_CONN_ALLOWED 4 (except for data connections)

TCP_DATA_CONN_ 8 (the control connection should be established or

ALLOWED allowed)

UDP_DATA_CONN_ 16 (the control connection should be established or

ALLOWED allowed)

The default value is 24, which is the sum of "TCP_DATA_CONN_ALLOWED" (value 8)

and UDP_DATA_CONN_ALLOWED (value 16). This means that the default allows only
TCP and UDP data connections to be opened under load.

ClusterXL R81 Administration Guide      |      143

 Defining Non-Monitored Interfaces

Defining Non-Monitored Interfaces

Non-Monitored interfaces are Cluster Member interfaces that are not monitored by the ClusterXL
mechanism.
You may wish to define an interface as non-monitored, if the interface is down for a long time, and you wish
the Cluster Member to continue to be Active.
1. Connect with SmartConsole to the Management Server.
2. From the left navigation panel, click Gateways & Servers .
3. Open the cluster object.
4. From the left tree, click Network Management.
5. Select the interface, which is down for a long time and click Edit.
6. In the Network Type section, select Private.
7. Click OK.
8. Install the Access Control Policy on this cluster object.

ClusterXL R81 Administration Guide      |      144

 Configuring Policy Update Timeout

Configuring Policy Update Timeout

When policy is installed on a cluster, the Cluster Members undertake a negotiation process to make sure
all of them have received the same policy before they actually apply it.
This negotiation process has a timeout mechanism, which makes sure a Cluster Member does not wait
indefinitely for responses from other Cluster Members, which is useful in cases when another Cluster
Member goes down when policy is being installed (for example).
In configurations, on which policy installation takes a long time (usually caused by a policy with a large
number of rules), a cluster with more than two members, and slow members, this timeout mechanism may
expire prematurely.
It is possible to tune the timeout with the kernel parameter fwha_policy_update_timeout_factor.
The default value of this kernel parameter is 1, which should be sufficient for most configurations.
For configurations where the situation described above occurs, setting the value of this kernel parameter
to 2 should be sufficient.

Warning - Do not set the value of this kernel parameter to a number larger than 3.

See the R81 Next Generation Security Gateway Guide - Chapter Working with Kernel Parameters on
Security Gateway.

ClusterXL R81 Administration Guide      |      145

 Enhanced 3-Way TCP Handshake Enforcement

Enhanced 3-Way TCP Handshake Enforcement

The standard enforcement for a 3-way handshake that initiates a TCP connection provides adequate
security by guaranteeing one-directional stickiness.
This means that it ensures that the SYN-ACK will always arrive after the SYN. However, it does not
guarantee that the ACK will always arrive after the SYN-ACK, or that the first data packet will arrive after the
ACK.
If you wish to have stricter policy that denies all out-of-state packets, you can configure the synchronization
mechanism so that all the TCP connection initiation packets arrive in the right sequence (SYN, SYN-ACK,
ACK, followed by the data).

Warning - The price for this extra security is a considerable delay in TCP connection
establishment.

Procedure to enable the enhanced TCP Handshake enforcement

1. Close all SmartConsole windows connected to the Management Server.

2. Connect with GuiDBedit Tool (see sk13009) to the Security Management Server or Domain
Management Server that manages this cluster.
3. In the left upper pane, go to Table > Network Objects > network_objects .
4. In the right upper pane, select the cluster object (the Class Name column shows gateway_
cluster).
5. Press the CTRL+F keys (or go to Search menu > Find).
6. In the Find window, paste this string and click Find Next:

sync_tcp_handshake_mode

7. In the lower pane, right-click on the sync_tcp_handshake_mode property and select Edit.
8. Choose complete_sync and click OK.
For more information, see the section "Synchronization modes for TCP 3-way handshake"
below.
9. To save the changes, from the File menu select Save All .
10. Close GuiDBedit Tool.
11. Connect with SmartConsole to the Security Management Server or Domain Management Server
that manages this cluster.
12. In SmartConsole, install the Access Control Policy onto the Cluster object.

ClusterXL R81 Administration Guide      |      146

 Enhanced 3-Way TCP Handshake Enforcement

Synchronization modes for TCP 3-way handshake

Mode Instructions

Minimal This is the default 3-way handshake synchronization mode.

sync The 3-way handshake is not enforced.
This mode offers the best connectivity for users who are willing to compromise on
security is this case.

Complete All 3-way handshake packets are Sync-and-ACK'ed, and the 3-way handshake is
sync enforced.
This mode slows down connection establishment considerably.
It may be used when there is no way to know where the next packet goes (for example,
in 3rd party clusters).

Smart In most cases, we can assume that if SYN and SYN-ACK were encountered by the
sync same cluster member, then the connection is “sticky”.
ClusterXL uses one additional flag in Connections Table record that says, “If this
member encounters a 3-way handshake packet, it should sync all other cluster
members”.
When a SYN packet arrives, the member that encountered it, records the connection
and turns off its flag. All other members are synchronized, and by using a post-sync
handler, their flag is turned on (in their Connections Tables).
If the same member encounters the SYN-ACK packet, the connection is sticky, thus
other cluster members are not informed.
Otherwise, the relevant member will inform all other member (since its flag is turned
on).
The original member (that encountered the SYN) will now turn on its flag, thus all
members will have their flag on.
In this case, the third packet of the 3-way handshake is also synchronized.
If for some reason, our previous assumption is not true (i.e., one cluster member
encountered both SYN and SYN-ACK packets, and other members encountered the
third ACK), then the “third” ACK will be dropped by the other cluster members, and we
rely on the periodic sync and TCP retransmission scheme to complete the 3-way
handshake.
This 3-way handshake synchronization mode is a good solution for ClusterXL Load
Sharing users that want to enforce 3-way handshake verification with the minimal
performance cost.
This 3-way handshake synchronization mode is also recommended for ClusterXL High
Availability.

ClusterXL R81 Administration Guide      |      147

 Cluster IP Addresses on Different Subnets

Cluster IP Addresses on Different Subnets

You can configure cluster Virtual IP addresses in different subnets than the physical IP addresses of the
Cluster Members.
The network "sees" the cluster as one Security Gateway that operates as a network router. The network is
not aware of the internal cluster structure and physical IP addresses of Cluster Members.

Advantages of using different subnets:

n You can create a cluster in an existing subnet that has a shortage of available IP addresses.
n You use only one Virtual IP address for the cluster. All other IP addresses can be on other subnets.
n You can "hide" physical Cluster Members' IP addresses behind the cluster Virtual IP address. This
security practice is almost the same as NAT.

Note - This capability is available only for ClusterXL clusters.

Traffic sent from Cluster Members to internal or external networks is hidden behind the cluster Virtual IP
addresses and cluster MAC addresses. The cluster MAC address assigned to cluster interfaces is:

Cluster Mode MAC Address

High Availability MAC address of the Active Cluster Member's interface

Load Sharing Multicast Multicast MAC address of the cluster Virtual IP Address

Load Sharing Unicast MAC address of the Pivot Cluster Member's interface

The use of different subnets with cluster objects has some limitations - see "Limitations of Cluster
Addresses on Different Subnets" on page 152.

ClusterXL R81 Administration Guide      |      148

 Configuring Cluster Addresses on Different Subnets

Configuring Cluster Addresses on Different Subnets

When using a cluster, in which the Cluster Virtual IP address and physical IP addresses of Cluster
Members are on different subnets, it is necessary to define the settings manually (see "Example of Cluster
IP Addresses on Different Subnets" on page 150).
1. Define these static routes for each cluster Virtual IP address

The required static routes must use applicable local member's interface as the next hop gateway
for network of cluster Virtual IP address.
If you do not define the static routes correctly, Cluster Members are not able to pass traffic.

Note - In VSX Cluster, you must configure all routes only in SmartConsole in
the VSX Cluster object.

For configuration instructions, see the R81 Gaia Administration Guide - Chapter Network
Management - Sections IPv4 Static Routes and IPv6 Static Routes.

2. Define the Cluster Members Network

a. In SmartConsole, use the Classic Mode to manually create a new cluster.

b. Define the Cluster Members and their physical interfaces.
c. From the left tree, click Network Management.
d. Select each cluster interface and click Edit.
e. In the General section, in the Virtual IPv4 field, enter the IPv4 address.
f. In the Member IPs , make sure the IP addresses are correct.
g. Click OK.
h. Install the Access Control Policy on this cluster object.
For more details, see the chapter "Configuring the Cluster Object and Members" on page 87.

ClusterXL R81 Administration Guide      |      149

 Example of Cluster IP Addresses on Different Subnets

Example of Cluster IP Addresses on Different Subnets

In this example, a cluster separates the network 172.16.6.0 / 24 (Side "A") from the network 172.16.4.0 /
24 (Side "B").
The Cluster Members use these IP addresses:
n 192.168.1.x / 24 for Side "A"
n 172.16.6.100 / 24 for Cluster Virtual IP Address for Side "A"
n 192.168.2.x / 24 for Side "B"
n 172.16.4.100 / 24 for Cluster Virtual IP Address for Side "B"
n 192.168.3.x / 24 for the Synchronization network

Procedure:

1. Configure static routes on the Cluster Members

On each Cluster Member, configure these static routes:

n Next hop gateway for network 172.16.6.0 is the local interface with the IP address
192.168.1.x
n Next hop gateway for network 172.16.4.0 is the local interface with the IP address
192.168.2.x
See the R81 Gaia Administration Guide > Chapter Network Management > Section IPv4 Static
Routes.

2. Configure Cluster IP Addresses in SmartConsole

a. Connect with SmartConsole to the Management Server.

b. From the left navigation panel, click Gateways & Servers .
c. Open the cluster object.
d. From the left tree, click Network Management.
e. Select each cluster interface and click Edit.

ClusterXL R81 Administration Guide      |      150

 Example of Cluster IP Addresses on Different Subnets

f. Configure these settings:

IP address of IP address of
Interface Properties
Cluster Interface "A" Cluster Interface "B

The General section - 172.16.6.100 / 24 172.16.4.100 / 24

the Virtual IPv4 field

The Member IPs section 192.168.1.1 / 24 192.168.2.1 / 24

192.168.1.2 / 24 192.168.2.2 / 24

g. Click OK.
h. Install the Access Control Policy on this cluster object.

ClusterXL R81 Administration Guide      |      151

 Limitations of Cluster Addresses on Different Subnets

Limitations of Cluster Addresses on Different Subnets

This new feature does not yet support all the capabilities of ClusterXL.
Some features require additional configuration to work properly, while others are not supported.

Connectivity Between Cluster Members

Since ARP requests issued by Cluster Members are hidden behind the cluster IP and MAC addresses,
requests sent by one Cluster Member to the other may be ignored by the destination computer.
To allow Cluster Members to communicate with each other, a static ARP should be configured for each
Cluster Member, stating the MAC addresses of all other Cluster Members. IP packets sent between
Cluster Members are not altered, and therefore no changes should be made to the routing table.

Note - Static ARP is not required in order for the Cluster Members to work properly as a
cluster, since the cluster synchronization protocol does not rely on ARP.

Load Sharing Multicast Mode with "Semi-Supporting" Hardware

Although not all types of network hardware work with multicast MAC addresses, some routers can pass
such packets, even though they are unable to handle ARP Replies containing a multicast MAC address.
Where a router semi-supports Load Sharing Multicast mode, it is possible to configure the cluster MAC
address as a static ARP entry in the router internal tables, and thus allow it to communicate with the cluster.
When different subnets are used for the cluster IP addresses, static ARP entries containing the router MAC
address need to be configured on each Cluster Member. This is done because this kind of router will not
respond to ARP Requests containing a multicast source MAC address. These special procedures are not
required when using routers that fully support multicast MAC addresses.

Manual Proxy ARP

When using Static NAT, the cluster can be configured to automatically recognize the hosts hidden behind
it, and issue ARP replies with the cluster MAC address, on their behalf. This process is known as Automatic
Proxy ARP.
However, if you use the ClusterXL VMAC mode or different subnets for the cluster IP addresses, this
mechanism will not work, and you must configure the proxy ARP manually. To do so, in SmartConsole,
click Menu > Global properties > NAT Network Address Translation, and disable Automatic ARP
Configuration. Then create the $FWDIR/conf/local.arp file.
For instructions, see sk30197.

Connecting to the Cluster Members from the Cluster Network

Because the unique IP addresses may be chosen arbitrarily, there is no guarantee that these addresses
are accessible from the subnet of the cluster IP address.
To access the Cluster Members through their unique IP addresses, you must configure routes on the
accessing Cluster Member, such that the cluster IP is the Default Gateway for the subnet of the unique IP
addresses.

ClusterXL R81 Administration Guide      |      152

 Limitations of Cluster Addresses on Different Subnets

Configuring Anti-Spoofing
1. Connect with SmartConsole to the Management Server.
2. From the left navigation panel, click Gateways & Servers .
3. Create a Group object, which contains the objects of both the external network and the internal
network.
In the "Example of Cluster IP Addresses on Different Subnets" on page 150, suppose Side "A" is the
external network, and Side "B" is the internal network.
You must configure the Group object to contain both the network 172.16.4.0 / 24 and the network
192.168.2.0 / 24.
4. Open the cluster object.
5. From the left tree, click Network Management.
6. Select the cluster interface and click Edit.
7. On the General page, in the Topology section, click Modify .
8. Select Override.
9. Select This Network (Internal).
10. Select Specific
11. Select the Group object that contains the objects of both the external network and the internal
network.
12. Click OK.
13. Install the Access Control Policy on this cluster object.

ClusterXL R81 Administration Guide      |      153

 Adding Another Member to an Existing Cluster

Adding Another Member to an Existing Cluster

Important - Schedule a full maintenance window to perform this procedure.

Best Practice - Before you change the current configuration, export a complete
management database with "migrate_server" command. See the R81 CLI
Reference Guide.

Adding a New Cluster Member to the Cluster Object

1. Install a new Cluster Member

Install a new Cluster Member you plan to add to the existing cluster.
See the R81 Installation and Upgrade Guide > Chapter Installing a ClusterXL, VSX Cluster,
VRRP Cluster.
Follow only the step "Install the Cluster Members".

Important - The new Cluster Member must run the same version with the
same Hotfixes as the existing Cluster Members.

2. Configure the new Cluster Member

On the new Cluster Member you plan to add to the existing cluster:
a. Configure or change the IP addresses on the applicable interfaces to match the current
cluster topology.
Use Gaia Portal or Gaia Clish.
See R81 Gaia Administration Guide.
b. Configure or change the applicable static routes to match the current cluster topology.
Use Gaia Portal or Gaia Clish.
c. Connect to the command line.
d. Log in to Gaia Clish or the Expert mode.
e. Start the Check Point Configuration Tool. Run:

cpconfig

f. Select the option Enable cluster membership for this gateway and enter y to confirm.
g. Reboot the new Cluster Member.

3. Configure the cluster object in SmartConsole

ClusterXL R81 Administration Guide      |      154

 Adding Another Member to an Existing Cluster

a. Connect with SmartConsole to the Security Management Server or Domain Management

Server that manages this cluster.
b. From the left navigation panel, click Gateways & Servers .
c. Open the existing cluster object.
d. In the Cluster Members page, click Add > New Cluster Member.
The Cluster Members Properties window opens.
Follow the instructions on the screen to configure this new Cluster Member.
e. Click the General tab.
f. In the Name field, enter a Cluster Member name.
g. In the IPv4 Address field, enter a physical IPv4 addresses.
The Management Server must be able to connect to the Cluster Member at this IPv4
address.
This IPv4 address can be an internal, or external. You can use a dedicated management
interface on the Cluster Member.

Important - You must define a corresponding IPv4 address for every

IPv6 address. This release does not support the configuration of only
IPv6 addresses.

h. In the IPv6 Address field, enter a physical IPv6 address, if you need to use IPv6.
The Management Server must be able to connect to the Cluster Member at this IPv6
address. This IPv6 address can be an internal, or external. You can use a dedicated
management interface on the Cluster Member.

Important - You must define a corresponding IPv4 address for every

IPv6 address. This release does not support the configuration of only
IPv6 addresses.

i. Click Communication, and initialize Secure Internal Communication (SIC) trust.

Enter the same key you entered during First Time Configuration Wizard on the new Cluster
Member.
j. Click the NAT tab to configure the applicable NAT settings.
k. Click the VPN tab to configure the applicable VPN settings.
l. Click OK.
m. From the left tree, click Network Management.
n Make sure all interfaces are defined correctly.
n Make sure all IP addresses are defined correctly.
n. Click OK.
o. Publish the SmartConsole session.

ClusterXL R81 Administration Guide      |      155

 Adding Another Member to an Existing Cluster

p. Install the Access Control Policy on this cluster object.

Policy installation must succeed on all Cluster Members.
q. Install the Threat Prevention Policy on this cluster object.
Policy installation must succeed on all Cluster Members.

4. Examine the cluster state

On each Cluster Member (existing and the newly added):

a. Connect to the command line.
b. Log in to Gaia Clish or the Expert mode.
c. Make sure all Cluster Members detect each other and agree on their cluster states. Run:

Shell Command

Gaia Clish i. set virtual-system

<VSID>
ii. show cluster state

Expert mode cphaprob [-vs <VSID>] state

For more information, see "Viewing Cluster State" on page 203.

If Cluster Members do not detect each other, or do not agree on their cluster states, then restart
the clustering.

Procedure

a. Connect to the command line on each Cluster Member (existing and the newly added).
b. Log in to Gaia Clish or the Expert mode.
c. Restart the clustering on each Cluster Member.
Run:

cphastop
cphastart

Important - This temporarily causes the Cluster Member not to be a

part of the cluster. As a result, cluster failover can occur.

ClusterXL R81 Administration Guide      |      156

 Adding Another Member to an Existing Cluster

d. Make sure all Cluster Members detect each other and agree on their cluster states.
Run:

Shell Command

Gaia Clish i. set virtual-system

<VSID>
ii. show cluster state

Expert mode cphaprob [-vs <VSID>] state

Adding an Existing Security Gateway as a Cluster Member

to the Cluster Object

Important - The existing Security Gateway must run the same version with the same
Hotfixes as the existing Cluster Members.

1. Configure the existing Security Gateway

On the existing Security Gateway you plan to add to the existing cluster:
a. Configure or change the IP addresses on the applicable interfaces to match the current
cluster topology.
Use Gaia Portal or Gaia Clish.
See R81 Gaia Administration Guide.
b. Configure or change the applicable static routes to match the current cluster topology.
Use Gaia Portal or Gaia Clish.
c. Connect to the command line.
d. Log in to Gaia Clish or the Expert mode.
e. Start the Check Point Configuration Tool. Run:

cpconfig

f. Select the option Enable cluster membership for this gateway and enter y to confirm.
g. Reboot the Security Gateway.

2. Configure the cluster object in SmartConsole

a. Connect with SmartConsole to the Security Management Server or Domain Management

Server that manages this cluster.
b. From the left navigation panel, click Gateways & Servers .
c. Open the existing cluster object.
d. From the left tree, click Cluster Members .

ClusterXL R81 Administration Guide      |      157

 Adding Another Member to an Existing Cluster

e. Click Add > Add Existing Gateway .

Follow the instructions on the screen to configure this new Cluster Member.
Read the warning and click Yes :

If you add <Name_of_Security_Gateway_object> to the

cluster, it will be converted to a cluster member. Some
settings will be lost.
The following settings will still remain:
-SIC
-VPN
-NAT (except for IP Pools)
In order to revert the conversion, session must be
discarded.
Are you sure you want to continue?

f. In the list of Cluster Members, select the new Cluster Member and click Edit.
g. Click the NAT tab and configure the applicable NAT settings.
h. Click the VPN tab and configure the applicable VPN settings.
i. From the left tree, click Network Management.
n Make sure all interfaces are defined correctly.
n Make sure all IP addresses are defined correctly.
j. Click OK.
k. Install the Access Control Policy on this cluster object.
Policy installation must succeed on all Cluster Members.
l. Install the Threat Prevention Policy on this cluster object.
Policy installation must succeed on all Cluster Members.

3. Examine the cluster state

On each Cluster Member (existing and the newly added):

a. Connect to the command line.
b. Log in to Gaia Clish or the Expert mode.

ClusterXL R81 Administration Guide      |      158

 Adding Another Member to an Existing Cluster

c. Make sure all Cluster Members detect each other and agree on their cluster states. Run:

Shell Command

Gaia Clish i. set virtual-system

<VSID>
ii. show cluster state

Expert mode cphaprob [-vs <VSID>] state

For more information, see "Viewing Cluster State" on page 203.

If Cluster Members do not detect each other, or do not agree on their cluster states, then restart
the clustering.

Procedure

a. Connect to the command line on each Cluster Member (existing and the newly added).
b. Log in to Gaia Clish or the Expert mode.
c. Restart the clustering on each Cluster Member.
Run:

cphastop
cphastart

Important - This temporarily causes the Cluster Member not to be a

part of the cluster. As a result, cluster failover can occur.

d. Make sure all Cluster Members detect each other and agree on their cluster states.
Run:

Shell Command

Gaia Clish i. set virtual-system

<VSID>
ii. show cluster state

Expert mode cphaprob [-vs <VSID>] state

ClusterXL R81 Administration Guide      |      159

 Removing a Member from an Existing Cluster

Removing a Member from an Existing Cluster

Important - Schedule a full maintenance window to perform this procedure.

Best Practice - Before you change the current configuration, export a complete
management database with "migrate_server" command. See the R81 CLI
Reference Guide.

1. Configure the cluster object in SmartConsole

a. Connect with SmartConsole to the Security Management Server or Domain Management

Server that manages this cluster.
b. From the left navigation panel, click Gateways & Servers .
c. Open the existing cluster object.
d. From the left tree, click Cluster Members page.
e. Click Remove > Delete Cluster Member.
Confirm when prompted.

Important:
n This operation deletes the object.
n There must be at least two Cluster Members in the cluster
object.

f. From the left tree, click Network Management.

n Make sure all interfaces are defined correctly.
n Make sure all IP addresses are defined correctly.
g. Click OK.
h. Install the Access Control Policy on the cluster object.

2. Restart the clustering and examine the cluster state

On each existing Cluster Member:

a. Connect to the command line.
b. Log in to Gaia Clish or the Expert mode.

ClusterXL R81 Administration Guide      |      160

 Removing a Member from an Existing Cluster

c. Restart the clustering.

Run:

cphastop
cphastart

Important - This temporarily causes the Cluster Member not to be a

part of the cluster. As a result, cluster failover can occur.

d. Make sure all Cluster Members detect each other and agree on their cluster states. Run:

Shell Command

Gaia Clish i. set virtual-system

<VSID>
ii. show cluster state

Expert mode cphaprob [-vs <VSID>] state

3. Configure the removed Cluster Member

On the Security Gateway you removed from the existing cluster:

a. Connect to the command line.
b. Log in to Gaia Clish or the Expert mode.
c. Start the Check Point Configuration Tool. Run:

cpconfig

d. Select the option Disable cluster membership for this gateway and enter y to confirm.
e. Select the option Secure Internal Communication > enter y to confirm > enter the new
Activation Key. Make sure to write it down.
f. Exit from the cpconfig menu.
g. Reboot the Security Gateway.

4. Establish SIC with the removed Security Gateway

If you need to use the Security Gateway you removed from the existing cluster, then establish
Secure Internal Communication (SIC) with it.
a. Connect with SmartConsole to the Security Management Server or Domain Management
Server that manages this Security Gateway.
b. From the left navigation panel, click Gateways & Servers .

ClusterXL R81 Administration Guide      |      161

 Removing a Member from an Existing Cluster

c. Create a new Security Gateway object in one of these ways:

n From the top toolbar, click the New ( ) > Gateway .

n In the top left corner, click Objects menu > More object types > Network Object >
Gateways and Servers > New Gateway .
n In the top right corner, click Objects Pane > New > More > Network Object >
Gateways and Servers > Gateway .
d. Follow the instructions on the screen.
Enter the same Activation Key you entered earlier in the cpconfig menu.

e. Click OK.
f. Publish the SmartConsole session.
g. Install the Access Control Policy on the Security Gateway object.
h. Install the Threat Prevention Policy on the Security Gateway object.

ClusterXL R81 Administration Guide      |      162

 ISP Redundancy on a Cluster

ISP Redundancy on a Cluster

In This Section:

Introduction 163
ISP Redundancy Modes 165
Outgoing Connections 165
Incoming Connections 166

Introduction
ISP Redundancy lets you connect Cluster Members to the Internet through redundant Internet Service
Provider (ISP) links.
ISP Redundancy monitors the ISP links and chooses the best current link.

Notes:
n R81 supports two ISPs.
n ISP Redundancy is intended for traffic that originates on your internal networks
and goes to the Internet.

Important:
n You must connect each Cluster Member with a dedicated physical interface to
each of the ISPs.
n The IP addresses assigned to physical interfaces on each Cluster Member must
be on the same subnet as the Cluster Virtual IP address.

IP addresses in the table below are only examples.

Item Description

1 Internal network

2 Switches

ClusterXL R81 Administration Guide      |      163

 ISP Redundancy on a Cluster

Item Description

3 Cluster Member A

3a Cluster interface connected to the internal network (IP address 10.10.10.0/24)

n Interface IP address 10.10.10.11
n Virtual IP address 10.10.10.1

3b Cluster interface (IP address 20.20.20.11) connected to the Sync network (IP address
20.20.20.0/24)

3c Cluster interface connected to a switch that connects to ISP A

n Interface IP address 30.30.30.11

n Virtual IP address 30.30.30.1

3d Cluster interface connected to a switch that connects to ISP B

n Interface IP address 40.40.40.11

n Virtual IP address 40.40.40.1

4 Cluster Member B

4a Cluster interface connected to the internal network (IP address 10.10.10.0/24)

n Interface IP address 10.10.10.22
n Virtual IP address 10.10.10.1

4b Cluster interface (IP address 20.20.20.22) connected to the Sync network (IP address
20.20.20.0/24)

4c Cluster interface connected to a switch that connects to ISP B

n Interface IP address 40.40.40.22

n Virtual IP address 40.40.40.1

4d Cluster interface connected to a switch that connects to ISP A

n Interface IP address 30.30.30.22

n Virtual IP address 30.30.30.1

5 ISP B

6 ISP A

7 Internet

ClusterXL R81 Administration Guide      |      164

 ISP Redundancy on a Cluster

ISP Redundancy Modes

ISP Redundancy configuration modes control the behavior of outgoing connections from internal clients to
the Internet:

Mode Description

Load Sharing Uses the two links to distribute load of connections.

Connections coming in are alternated.
You can configure best relative loads for the links (set a faster link to handle more
load).
New connections are randomly assigned to a link.
If one link fails, the other link takes the load.
In this mode, incoming connections can reach the application servers through
either ISP link because the Cluster can answer DNS requests for the IP address of
internal servers with IP addresses from both ISPs by alternating their order.

Primary/Backup Uses one link for connections.

It switches to the Backup link if the Primary link fails.
When the Primary link is restored, new connections are assigned to it.
Existing connections continue on the Backup link until they are complete.
In this mode, incoming connections (from the Internet to application servers in the
DMZ or internal networks) also benefit, because the Cluster returns packets using
the same ISP Link, through which the connection was initiated.

Best Practice:
n If both ISPs are basically the same, use the Load Sharing mode to ensure that
you are making the best use of both ISPs.
n You may prefer to use one of your two ISPs that is more cost-effective in terms of
price and reliability. In that case, use Primary/Backup mode and set the more
cost-effective ISP as the Primary ISP link.

Outgoing Connections
n In ISP Redundancy Load Sharing mode, outgoing traffic that exits the Cluster on its way to the
Internet is distributed between the ISP Links. You can set a relative weight for how much you want
each of the ISP Links to be used.
For example, if one link is faster, it can be configured to route more traffic across that ISP link than
the other.
n In ISP Redundancy Primary/Backup mode, outgoing traffic uses an active primary link.
Hide NAT is used to change the source address of outgoing packets to the address of the interface,
through which the packet leaves the Cluster. This allows return packets to be automatically routed
through the same ISP link, because their destination address is the address of the correct link. Hide
NAT is configured by the administrator.

ClusterXL R81 Administration Guide      |      165

 ISP Redundancy on a Cluster

Incoming Connections
For external users to make incoming connections, the administrator must give each application server two
routable IP addresses, one for each ISP. The administrator must also configure Static NAT to translate the
routable addresses to the real server address.
If the servers handle different services (for example, HTTP and FTP), you can use NAT to employ only two
routable IP addresses for all the publicly available servers.
External clients use one of the two addresses. In order to connect, the clients must be able to resolve the
DNS name of the server to the correct IP address.

Note - In the following example, the subnets 172.16.0.0/24 and 192.168.0.0/24

represent public routable addresses.

In the following example, the Web server www.example.com is assigned an IP address from each ISP:
n 192.168.1.2 from ISP A
n 172.16.2.2 from ISP B
If the ISP Link A is down, then IP address 192.168.1.2 becomes unavailable, and the clients must be able
to resolve the URL www.example.com to the IP address 172.16.2.2.
An incoming connection is established, based on this example, in the following sequence:
1. When an external client on the Internet contacts www.example.com , the client sends a DNS query
for the IP address of this URL.
The DNS query reaches the Cluster. The Cluster has a built-in mini-DNS server that can be
configured to intercept DNS queries (of Type A) for servers in its domain.
2. A DNS query arriving at an interface that belongs to one of the ISP links, is intercepted by the
Cluster.
3. If the Cluster recognizes the name of the host, it sends one of the following replies:
n In ISP Redundancy Primary/Backup mode, the Cluster replies only with the IP addresses
associated with the Primary ISP link, as long as the Primary ISP link is active.
n In ISP Redundancy Load Sharing mode, the Cluster replies with two IP addresses,
alternating their order.
4. If the Cluster is unable to handle DNS requests (for example, it may not recognize the host name), it
passes the DNS query to its original destination or the DNS server of the domain example.com .
5. When the external client receives the reply to its DNS query, it opens a connection. Once the
packets reach the Cluster, the Cluster uses Static NAT to translate the destination IP address
192.168.1.2 or 172.16.2.2 to the real server IP address 10.0.0.2.
6. The Cluster routes the reply packets from the server to the client through the same ISP link that was
used to initiate the connection.

ClusterXL R81 Administration Guide      |      166

 Configuring ISP Redundancy on a Cluster

Configuring ISP Redundancy on a Cluster

1. Connect with SmartConsole to the Security Management Server or Domain Management Server
that manages this Cluster.
2. From the left navigation panel, click Gateways & Servers .
3. Open the Cluster object.
4. Click Other > ISP Redundancy .
5. Select Support ISP Redundancy .
6. Select the redundancy mode - Load Sharing or Primary/Backup.
7. Configure the ISP Links.
Procedure

Make sure you have the ISP data - the speed of the link and next hop IP address.
Automatic vs Manual configuration:
n If the Cluster object has two interfaces with Topology "External " in the Network
Management page, you can configure the ISP links automatically.
Configuring ISP links automatically

a. Click Other > ISP Redundancy .

b. Click Set initial configuration.
The ISP Links are added automatically.
c. For Primary/Backup mode, make sure the Primary interface is first in the list. Use
the arrows on the right to change the order.
d. Click OK.

ClusterXL R81 Administration Guide      |      167

 Configuring ISP Redundancy on a Cluster

n If the Cluster object only one interface with Topology "External " in the Network
Management page, you must configure the ISP links manually.
Configuring ISP links manually

a. Click Other > ISP Redundancy .

b. In the IPS Links section, click Add.
The ISP Link window opens.
c. Click the General tab.
d. In the Name field, enter a name of this link (desired text).
The name you enter here is used in the ISP Redundancy commands (see
"Controlling ISP Redundancy from CLI" on page 173).
e. Select the Interface of the Cluster for this ISP link.
l If the Cluster object has two interfaces with Topology "External " in the
Network Management page, set each ISP link to a different interface.
If one of the ISP links is the connection to a backup ISP, configure the ISP
Redundancy Script (see "Controlling ISP Redundancy from CLI" on
page 173).
l If the Cluster object only one interface with Topology "External " in the
Network Management page, set each ISP link to connect to this interface.
f. Configure the Next Hop IP Address .
l If the Cluster object has two interfaces with Topology "External " in the
Network Management page, leave this field empty and click Get from
routing table. The next hop is the default gateway.
l If the Cluster object only one interface with Topology "External " in the
Network Management page, set each ISP link to a different next hop
router.
g. For ISP Redundancy in Load Sharing mode, enter the Weight value.
For equal traffic distribution between the two IPS link, enter 50 in each ISP link.
If one ISP link is faster, increase this value and decrease it for the other ISP link,
so that the sum of these two values is always equal 100.
h. Click the Advanced tab.
i. Define hosts to be monitored, to make sure the link is working.
Add the applicable objects to the Selected hosts section.
j. Click OK.

8. Configure the Cluster to be the DNS server.

Procedure

The Cluster, or a DNS server behind it, must respond to DNS queries.

ClusterXL R81 Administration Guide      |      168

 Configuring ISP Redundancy on a Cluster

It resolves IP addresses of servers in the DMZ (or another internal network).

Get a public IP address from each ISP.
If public IP addresses are not available, register the domain to make the DNS server accessible
from the Internet.
The Cluster intercepts DNS queries "Type A" for the web servers in its domain that come from
external hosts.
n If the Cluster recognizes the external host, it replies:
l In ISP Redundancy Load Sharing mode, the Cluster replies with two IP addresses,
alternating their order.
l In ISP Redundancy Primary/Backup mode, the Cluster replies with the IP
addresses of the active ISP link.
n If the Cluster does not recognize the host, it passes the DNS query on to the original
destination, or to the domain DNS server.

To enable DNS server:

a. Click Other > ISP Redundancy .
b. Select Enable DNS Proxy .
c. Click Configure.
d. Add your DMZ or Web servers. Give each server two public IP addresses - one from each
ISP.
e. In the DNS TTL, enter a number of seconds.
This sets a Time To Live for each DNS reply.
DNS servers in the Internet cannot cache your DNS data in the reply for longer than the
TTL.
f. Click OK.
g. Configure Static NAT to translate the public IP addresses to the real server's IP address.
External clients use one of the two IP addresses.

Note - If the servers use different services (for example, HTTP and
FTP), you can use NAT for only two public IP addresses.

h. Define an Access Control Policy rule:

Services & Install

Name Source Destination VPN Action Track
Applications On

DNS Applicable Applicable DNS Any domain_udp Accept None Policy

Proxy sources Servers Targets

ClusterXL R81 Administration Guide      |      169

 Configuring ISP Redundancy on a Cluster

To register the domain and get IP addresses:

a. Register your domain with the two ISP.
b. Tell the ISP the two IP addresses of the DNS server that respond to DNS queries for the
domain.
c. For each server in the DMZ, get two public IP addresses, one from each ISP.
d. In SmartConsole, click Menu > Global properties .
e. From the left tree, click NAT - Network Address Translation.
f. In the Manual NAT rules section, select Translate destination on client side.
g. Click OK.

9. Configure the Access Control Policy for ISP Redundancy.

Procedure

The Access Control Policy must allow connections through the ISP links, with Automatic Hide NAT
on network objects that start outgoing connections.
a. In the properties of the object for an internal network, select NAT > Add Automatic
Address Translation Rules .
b. Select Hide behind the gateway .
c. Click OK.

ClusterXL R81 Administration Guide      |      170

 Configuring ISP Redundancy on a Cluster

d. Define rules for publicly reachable servers (Web servers, DNS servers, DMZ servers).
n If you have one public IP address from each ISP for the Cluster, define Static NAT.
Allow specific services for specific servers.
For example, make NAT rules, so that incoming HTTP connections from the two
ISPs reach a Web server, and DNS traffic from the ISP reach the DNS server.
Example: Manual Static Rules for a Web Server and a DNS Server

Origi Origin Translat Translat

Original Translat Inst
nal al ed ed Comm
Destinat ed all
Sour Servic Destinat Service ent
ion Source On
ce es ion s

Any Host object http = S 50.50.50.2 = Polic Incoming

with IP Original Original y Web - ISP
address of Targe A
Web Server
ts

Any Host object http = S 60.60.60.2 = Polic Incoming

with IP Original Original y Web - ISP
address of Targe B
Web Server
ts

Any Host object domain_ = S 50.50.50.3 = Polic Incoming

with IP udp Original Original y DNS - ISP
address of Targe A
DNS Server
ts

Any Host object domain_ = S 60.60.60.3 = Polic Incoming

with IP udp Original Original y DNS - ISP
address of Targe B
DNS Server
ts

n If you have a public IP address from each ISP for each publicly reachable server (in
addition to the Cluster), define NAT rules:
i. Give each server a private IP address.
ii. Use the public IP addresses in the Original Destination.
iii. Use the private IP address in the Translated Destination.
iv. Select Any as the Original Service.

Note - If you use Manual NAT, then automatic ARP does not work for the IP
addresses behind NAT. You must configure the local.arp file as
described in sk30197.

10. Install the Access Control Policy on this Cluster object.

ClusterXL R81 Administration Guide      |      171

 ISP Redundancy and VPN

ISP Redundancy and VPN

Note - ISP Redundancy settings override the VPN Link Selection settings.

When ISP Redundancy is enabled, VPN encrypted connections survive a failure of an ISP link.
The settings in the ISP Redundancy page override settings in the IPsec VPN > Link Selection page.

Configuring ISP Redundancy for VPN with a Check Point peer

Step Instructions

1 Connect with SmartConsole to the Security Management Server or Domain Management

Server that manages this Cluster.

2 From the left navigation panel, click Gateways & Servers .

3 Open the Cluster object.

4 In the left navigation tree, go to Other > ISP Redundancy .

5 Select Apply settings to VPN traffic .

6 In the left navigation tree, go to IPsec VPN > Link Selection.

7 Make sure that Use ongoing probing. Link redundancy mode shows the mode of the ISP
Redundancy:
High Availability (for Primary/Backup) or Load Sharing.
The VPN Link Selection now only probes the ISP configured in ISP Redundancy.

Configuring ISP Redundancy for VPN with a third-party peer

If the VPN peer is not a Check Point Security Gateway, the VPN may fail, or the third-party device may
continue to encrypt traffic to a failed ISP link.
n Make sure the third-party VPN peer recognizes encrypted traffic from the secondary ISP link as
coming from the Check Point cluster.
n Change the configuration of ISP Redundancy to not use these Check Point technologies:
l Use Probing - Makes sure that Link Selection uses another option.
l The options Load Sharing, Service Based Link Selection, and Route based probing
work only on Check Point Security Gateways and Clusters.
If used, the Security Gateway or Cluster Members use one link to connect to the third-party
VPN peer.
The link with the highest prefix length and lowest metric is used.

ClusterXL R81 Administration Guide      |      172

 Controlling ISP Redundancy from CLI

Controlling ISP Redundancy from CLI

You can control the ISP Redundancy behavior from CLI.

Force ISP Link State

Use the "fw isp_link" command to force the ISP link state to Up or Down.
Use this to test installation and deployment, or to force the Cluster Members to recognize the true link state
if it cannot (the ISP link is down but the gateway sees it as up).
n You can run this command on the Cluster Members:

fw isp_link <Name of ISP Link in SmartConsole> {up | down}

n You can run this command on the Security Management Server:

fw isp_link <Name of Cluster Member Object> <Name of ISP Link in

SmartConsole> {up | down}

For more information, see the R81 CLI Reference Guide > Chapter Security Gateway Commands -
Section fw - Section fw isp_link.

The ISP Redundancy Script

When the Cluster Member starts, or an ISP link state changes, the $FWDIR/bin/cpisp_update script
runs on the Cluster Member.
This script changes the default route of the Cluster Member.
For example, you can force the Cluster Member to change the state of a dialup interface to match that
state of its ISP link.
Edit this script to enable a dialup connection for one of the ISP links.

To configure a dialup connection:

1. In the script on the Cluster Member, enter the command to change the dialup interface state:
n If the ISP link goes down:

fw isp_link <Name of ISP Link in SmartConsole> down

n If the ISP link goes up:

fw isp_link <Name of ISP Link in SmartConsole> up

2. If you use PPPoE or PPTP xDSL modems, in the PPPoE or PPTP configuration, the Use Peer as
Default Gateway option must be cleared.

ClusterXL R81 Administration Guide      |      173

 Dynamic Routing Protocols in a Cluster Deployment

Dynamic Routing Protocols in a Cluster

Deployment
ClusterXL supports Dynamic Routing (Unicast and Multicast) protocols as an integral part of Gaia
Operating System.
As the network infrastructure views the clustered Security Gateway as a single logical entity, failure of a
Cluster Member will be transparent to the network infrastructure and will not result in a ripple effect.

Router IP Address

All Cluster Members use the Cluster Virtual IP address(es) as Router IP address(es).

Routing Table Synchronization

Routing information is synchronized among the Cluster Members using the Forwarding Information Base
(FIB) Manager process.
This is done to prevent traffic interruption in case of failover, and used for Load Sharing and High
Availability modes.
The FIB Manager is the responsible for the routing information.
The FIB Manager is registered as a Critical Device called "FIB". If the slave goes out of sync, this Critical
Device reports its state as "problem". As a result, the slave member changes its state to "DOWN" until the
FIB Manager is synchronized.

Wait for Clustering

n When Dynamic Routing protocols and/or DHCP Relay are configured on cluster, the "Wait for
Clustering" option must be enabled in these cluster modes:
l ClusterXL High Availability
l ClusterXL Load Sharing Unicast
l ClusterXL Load Sharing Multicast
l VSX High Availability
l VSX Load Sharing (VSLS)
n When Dynamic Routing protocols and/or DHCP Relay are configured on cluster, the "Wait for
Clustering" must be disabled in these cluster modes:
l VRRP Cluster on Gaia OS
For more information, see sk92322.

ClusterXL R81 Administration Guide      |      174

 Dynamic Routing Protocols in a Cluster Deployment

Failure Recovery

Dynamic Routing on ClusterXL avoids creating a ripple effect upon failover by informing the neighboring
routers that the router has exited a maintenance mode.
The neighboring routers then reestablish their relationships to the cluster, without informing the other
routers in the network.
These restart protocols are widely adopted by all major networking vendors.
This table lists the RFC and drafts compliant with Check Point Dynamic Routing:

Protocol RFC or Draft

OSPF Graceful restart RFC 3623

OSPF LLS RFC 5613

BGP Graceful restart RFC 4724

ClusterXL R81 Administration Guide      |      175

 ClusterXL Configuration Commands

ClusterXL Configuration Commands

Description
These commands let you configure internal behavior of the Clustering Mechanism.

Important:
n We do not recommend that you run these commands. These commands must
be run automatically only by the Security Gateway or the Check Point Support.
n In a Cluster, you must configure all the Cluster Members in the same way.

Syntax

Notes:
n In Gaia Clish:
Enter the set cluster<ESC><ESC> to see all the available commands.
n In Expert mode:
Run the cphaconf command see all the available commands.
You can run the cphaconf commands only from the Expert mode.
n Syntax legend:
1. Curly brackets or braces { }:
Enclose a list of available commands or parameters, separated by the
vertical bar |, from which user can enter only one.
2. Angle brackets < > :
Enclose a variable - a supported value user needs to specify explicitly.
3. Square brackets or brackets [ ]:
Enclose an optional command or parameter, which user can also enter.
n You can include these commands in scripts to run them automatically.
The meaning of each command is explained in the next sections.
Table: ClusterXL Configuration Commands
Description Command in Command in
of Command Gaia Clish Expert Mode

Configure how to show the Cluster Member in set cphaconf mem_id_mode {id
local ClusterXL logs - by its Member ID or its cluster | name}
Member Name (see "Configuring the Cluster member
Member ID Mode in Local Logs" on page 180) idmode {id
| name}

Register a single Critical Device (Pnote) on the N / A cphaconf set_pnote -d

Cluster Member (see "Registering a Critical <Name of Device> -t
Device" on page 181) <Timeout in Sec> -s
{ok|init|problem} [-p]
[-g] register

ClusterXL R81 Administration Guide      |      176

 ClusterXL Configuration Commands

Table: ClusterXL Configuration Commands (continued)

Description Command in Command in
of Command Gaia Clish Expert Mode

Unregister a single Critical Device (Pnote) on N / A cphaconf set_pnote -d

the Cluster Member (see "Unregistering a <Name of Device> [-p] [-
Critical Device" on page 183) g] unregister

Report (change) a state in a single Critical N / A cphaconf set_pnote -d

Device (Pnote) on the Cluster Member (see <Name of Device> -s
"Reporting the State of a Critical Device" on {ok|init|problem} [-g]
page 184) report

Register several Critical Devices (Pnotes) from N / A cphaconf set_pnote -f

a file on the Cluster Member (see "Registering <Name of File> [-g]
Critical Devices Listed in a File" on page 185) register

Unregister all Critical Devices (Pnotes) on the N / A cphaconf set_pnote -a [-

Cluster Member (see "Unregistering All Critical g] unregister
Devices" on page 187)

Configure the Cluster Control Protocol (CCP) set cphaconf ccp_encrypt

Encryption on the Cluster Member (see cluster {off | on}
"Configuring the Cluster Control Protocol member cphaconf ccp_encrypt_key
(CCP) Settings" on page 188) ccpenc <Key String>
{off | on}

Configure the Cluster Forwarding Layer on the set cphaconf forward {off |
Cluster Member (controls the forwarding of cluster on}
traffic between Cluster Members) member
Note - For Check Point use only. forwarding
{off | on}

Print the current cluster configuration as loaded N / A cphaconf debug_data

in the kernel on the Cluster Member (for details,
see sk93306)

Start internal failover between slave interfaces N / A cphaconf failover_bond

of specified bond interface - only in Bond High <bond_name>
Availability mode (for details, see sk93306)

Configure what happens during a failover after N / A cphaconf enable_bond_

a Bond already failed over internally (for details, failover <bond_name>
see sk93306)

Initiate manual cluster failover (see "Initiating set clusterXL_admin {down |

Manual Cluster Failover" on page 189) cluster up}
member
admin
{down |
up}

ClusterXL R81 Administration Guide      |      177

 ClusterXL Configuration Commands

Table: ClusterXL Configuration Commands (continued)

Description Command in Command in
of Command Gaia Clish Expert Mode

Configure the minimal number of required N / A cphaconf bond_ls {set

slaves interfaces for Bond Load Sharing (see <Bond Name> <Value> |
"Configuring the Minimal Number of Required remove <Bond Name>}
Slave Interfaces for Bond Load Sharing" on
page 193)

Configuring Link Monitoring on the Cluster N / A N / A

Interfaces (see "Configuring Link Monitoring on
the Cluster Interfaces" on page 107)

Configuring the Multi-Version Cluster N / A cphaconf mvc {off | on}

Mechanism (see "Configuring the Multi-
Version Cluster Mechanism" on page 196)

List of the Gaia Clish set cluster member commands

set cluster member admin {down | up} [permanent]

set cluster member ccpenc {off | on}
set cluster member forwarding {off | on}
set cluster member idmode {id | name}
set cluster member mvc {off | on}

ClusterXL R81 Administration Guide      |      178

 ClusterXL Configuration Commands

List of the cphaconf commands

Note - Some commands are not applicable to 3rd party clusters.

cphaconf [-D] <options> start

cphaconf stop
cphaconf [-t <Sync IF 1>...] [-d <Non-Monitored IF 1>...] add
cphaconf clear-secured
cphaconf clear-non-monitored
cphaconf debug_data
cphaconf delete_link_local [-vs <VSID>] <IF name>
cphaconf set_link_local [-vs <VSID>] <IF name> <Cluster IP>
cphaconf mem_id_mode {id | name}
cphaconf failover_bond <bond_name>
cphaconf [-s] {set | unset | get} var <Kernel Parameter Name>
[<Value>]
cphaconf bond_ls {set <Bond Name> <Value> | remove <Bond Name>}
cphaconf set_pnote -d <Device> -t <Timeout in sec> -s {ok | init |
problem} [-p] [-g] register
cphaconf set_pnote -f <File> [-g] register
cphaconf set_pnote -d <Device> [-p] [-g] unregister
cphaconf set_pnote -a [-g] unregister
cphaconf set_pnote -d <Device> -s {ok | init | problem} [-g] report
cphaconf ccp_encrypt {off | on}
cphaconf ccp_encrypt_key <Key String>

ClusterXL R81 Administration Guide      |      179

 Configuring the Cluster Member ID Mode in Local Logs

Configuring the Cluster Member ID Mode in

Local Logs

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
This command lets you configure how to show the Cluster Member in the local ClusterXL logs - by its
Member ID (default), or its Member Name.
This configuration affects these local logs:
n /var/log/messages
n dmesg
n $FWDIR/log/fwd.elg
See "Viewing the Cluster Member ID Mode in Local Logs" on page 235.

Syntax

Shell Command

Gaia Clish set cluster member idmode {id | name}

Expert mode cphaconf mem_id_mode {id | name}

Example

[Expert@Member1:0]# cphaprob names

Current member print mode in local logs is set to: ID

[Expert@Member1:0]#
[Expert@Member1:0]# cphaconf mem_id_mode name

Member print mode in local logs: NAME

[Expert@Member1:0]#
[Expert@Member1:0]# cphaprob names

Current member print mode in local logs is set to: NAME

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      180

 Registering a Critical Device

Registering a Critical Device

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
You can add a user-defined critical device to the default list of critical devices. Use this command to
register <device> as a critical process, and add it to the list of devices that must run for the Cluster Member
to be considered active. If <device> fails, then the Cluster Member is seen as failed.
If a Critical Device fails to report its state to the Cluster Member in the defined timeout, the Critical Device,
and by design the Cluster Member, are seen as failed.
Define the status of the Critical Device that is reported to ClusterXL upon registration.
This initial status can be one of these:
n ok - Critical Device is alive.
n init - Critical Device is initializing. The Cluster Member is Down. In this state, the Cluster Member
cannot become Active.
n problem - Critical Device failed. If this state is reported to ClusterXL, the Cluster Member
immediately goes Down. This causes a failover.

Syntax

Shell Command

Gaia N/A
Clish

Expert cphaconf set_pnote -d <Name of Critical Device> -t <Timeout in

mode Sec> -s {ok | init | problem} [-p] [-g] register

Notes:
n The "-t" flags specifies how frequently to expect the periodic reports from this Critical
Device.
If no periodic reports should be expected, then enter the value 0 (zero).
n The "-p" flag makes these changes permanent (survive reboot).
n The "-g" flag applies the command to all configured Virtual Systems.

Restrictions
n Total number of critical devices (pnotes) on Cluster Member is limited to 16.
n Name of any critical device (pnote) on Cluster Member is limited to 15 characters, and must not
include white spaces.

ClusterXL R81 Administration Guide      |      181

 Registering a Critical Device

Related topics
n "Viewing Critical Devices" on page 207
n "Reporting the State of a Critical Device" on page 184
n "Registering Critical Devices Listed in a File" on page 185
n "Unregistering a Critical Device" on page 183
n "Unregistering All Critical Devices" on page 187

ClusterXL R81 Administration Guide      |      182

 Unregistering a Critical Device

Unregistering a Critical Device

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
This command lets you unregister a user-defined Critical Device (Pnote). This means that this device is no
longer considered critical.
If a Critical Device was registered with a state "problem", before you ran this command, then after you
run this command, the status of the Cluster Member depends only on the states of the remaining Critical
Devices.

Syntax

Shell Command

Gaia Clish N/A

Expert cphaconf set_pnote -d <Name of Critical Device> [-p] [-g]

mode unregister

Notes:
n The "-p" flag makes these changes permanent.
This means that after you reboot, these Critical Devices remain
unregistered.
n The "-g" flag applies the command to all configured Virtual Systems.

Related topics
n "Viewing Critical Devices" on page 207
n "Reporting the State of a Critical Device" on page 184
n "Registering a Critical Device" on page 181
n "Registering Critical Devices Listed in a File" on page 185
n "Unregistering All Critical Devices" on page 187

ClusterXL R81 Administration Guide      |      183

 Reporting the State of a Critical Device

Reporting the State of a Critical Device

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
This command lets you report (change) manually the state of a Critical Device to ClusterXL.
The reported state can be one of these:
n ok - Critical Device is alive.
n init - Critical Device is initializing. The Cluster Member is Down. In this state, the Cluster Member
cannot become Active.
n problem - Critical Device failed. If this state is reported to ClusterXL, the Cluster Member
immediately goes Down. This causes a failover.
If a Critical Device fails to report its state to the Cluster Member within the defined timeout, the Critical
Device, and by design the Cluster Member, are seen as failed. This is true only for Critical Devices with
timeouts. If a Critical Device is registered with the "-t 0" parameter, there is no timeout. Until the Critical
Device reports otherwise, the state of the Critical Device is considered to be the last reported state.

Syntax

Shell Command

Gaia N/A
Clish

Expert cphaconf set_pnote -d <Name of Critical Device> -s {ok | init |

mode problem} [-g] report

Notes:
n The "-g" flag applies the command to all configured Virtual Systems.
n If the "<Name of Critical Device>" reports its state as "problem", then
the Cluster Member reports its state as failed.

Related topics
n "Viewing Critical Devices" on page 207
n "Registering a Critical Device" on page 181
n "Registering Critical Devices Listed in a File" on page 185
n "Unregistering a Critical Device" on page 183
n "Unregistering All Critical Devices" on page 187

ClusterXL R81 Administration Guide      |      184

 Registering Critical Devices Listed in a File

Registering Critical Devices Listed in a File

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
This command lets you register all the user-defined Critical Devices listed in the specified file.
This file must be a plain-text ASCII file, with each Critical Device defined on a separate line.
Each definition must contain three parameters, which must be separated by a space or a tab character:

<Name of Device> <Timeout> <Status>

Where:

Parameter Description

<Name The name of the Critical Device.

of
n Maximal name length is 15 characters
Device>
n The name must not include white spaces (space or tab characters).

< If the Critical Device <Name of Device> fails to report its state to the Cluster Member
Timeout within this specified number of seconds, the Critical Device (and by design the Cluster
> Member), are seen as failed.
For no timeout, use the value 0 (zero).

< The Critical Device <Name of Device> reports one of these statuses to the Cluster
Status> Member:
n ok - Critical Device is alive.
n init- Critical Device is initializing. The Cluster Member is Down. In this state, the
Cluster Member cannot become Active.
n problem - Critical Device failed. If this state is reported to ClusterXL, the Cluster
Member immediately goes Down. This causes a failover.

Syntax

Shell Command

Gaia Clish N/A

Expert mode cphaconf set_pnote -f /<Path>/<Name of File> [-g] register

Note - The "-g" flag applies the command to all configured Virtual Systems.

ClusterXL R81 Administration Guide      |      185

 Registering Critical Devices Listed in a File

Related topics
n "Viewing Critical Devices" on page 207
n "Reporting the State of a Critical Device" on page 184
n "Registering a Critical Device" on page 181
n "Unregistering a Critical Device" on page 183
n "Unregistering All Critical Devices" on page 187

ClusterXL R81 Administration Guide      |      186

 Unregistering All Critical Devices

Unregistering All Critical Devices

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
This command lets you unregister all critical devices from the Cluster Member.

Syntax

Shell Command

Gaia Clish N/A

Expert mode cphaconf set_pnote -a [-g] unregister

Notes:
n The "-a" flag specifies that all Pnotes must be unregistered
n The "-g" flag applies the command to all configured Virtual
Systems

Related topics
n "Viewing Critical Devices" on page 207
n "Reporting the State of a Critical Device" on page 184
n "Registering a Critical Device" on page 181
n "Registering Critical Devices Listed in a File" on page 185
n "Unregistering a Critical Device" on page 183

ClusterXL R81 Administration Guide      |      187

 Configuring the Cluster Control Protocol (CCP) Settings

Configuring the Cluster Control Protocol (CCP)

Settings

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
Cluster Members configure the Cluster Control Protocol (CCP) mode automatically.

Important - In R81, the CCP always runs in the unicast mode.

You can configure the Cluster Control Protocol (CCP) Encryption on the Cluster Members.
See "Viewing the Cluster Control Protocol (CCP) Settings" on page 240.

Syntax for configuring the Cluster Control Protocol (CCP) Encryption

Shell Command

Gaia Clish set cluster member ccpenc {off | on}

Expert mode cphaconf ccp_encrypt {off | on}

cphaconf ccp_encrypt_key <Key String>

ClusterXL R81 Administration Guide      |      188

 Initiating Manual Cluster Failover

Initiating Manual Cluster Failover

Description
This command lets you initiate a manual cluster failover (see sk55081 and "How to Initiate Cluster
Failover" on page 250).

Syntax

Shell Command

Gaia Clish set cluster member admin {down | up}

Expert mode clusterXL_admin {down | up}

ClusterXL R81 Administration Guide      |      189

 Initiating Manual Cluster Failover

Example

ClusterXL R81 Administration Guide      |      190

 Initiating Manual Cluster Failover

[Expert@Member1:0]# cphaprob state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 (local) 11.22.33.245 100% ACTIVE Member1

2 11.22.33.246 0% STANDBY Member2

Active PNOTEs: None

... ...

[Expert@Member1:0]#

[Expert@Member1:0]# clusterXL_admin down

This command does not survive reboot. To make the change permanent, please run 'set cluster member admin
down/up permanent' in clish or add '-p' at the end of the command in expert mode
Setting member to administratively down state ...
Member current state is DOWN
[Expert@Member1:0]#

[Expert@Member1:0]# cphaprob state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 (local) 11.22.33.245 0% DOWN Member1

2 11.22.33.246 100% ACTIVE Member2

Active PNOTEs: ADMIN

Last member state change event:

Event Code: CLUS-111400
State change: ACTIVE -> DOWN
Reason for state change: ADMIN_DOWN PNOTE
Event time: Sun Sep 8 19:35:06 2019

Last cluster failover event:

Transition to new ACTIVE: Member 1 -> Member 2
Reason: ADMIN_DOWN PNOTE
Event time: Sun Sep 8 19:35:06 2019

Cluster failover count:

Failover counter: 2
Time of counter reset: Sun Sep 8 16:08:34 2019 (reboot)

[Expert@Member1:0]#

[Expert@Member1:0]# clusterXL_admin up
This command does not survive reboot. To make the change permanent, please run 'set cluster member admin
down/up permanent' in clish or add '-p' at the end of the command in expert mode
Setting member to normal operation ...
Member current state is STANDBY
[Expert@Member1:0]#

[Expert@Member1:0]# cphaprob state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 (local) 11.22.33.245 0% STANDBY Member1

2 11.22.33.246 100% ACTIVE Member2

Active PNOTEs: None

Last member state change event:

ClusterXL R81 Administration Guide      |      191

 Initiating Manual Cluster Failover

Event Code: CLUS-114802

State change: DOWN -> STANDBY
Reason for state change: There is already an ACTIVE member in the cluster (member 2)
Event time: Sun Sep 8 19:37:03 2019

Last cluster failover event:

Transition to new ACTIVE: Member 1 -> Member 2
Reason: ADMIN_DOWN PNOTE
Event time: Sun Sep 8 19:35:06 2019

Cluster failover count:

Failover counter: 2
Time of counter reset: Sun Sep 8 16:08:34 2019 (reboot)

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      192

 Configuring the Minimal Number of Required Slave Interfaces for Bond Load Sharing

Configuring the Minimal Number of Required

Slave Interfaces for Bond Load Sharing

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Description
ClusterXL considers a bond in Load Sharing mode to be in the "down" state when fewer than a minimal
number of required slave interfaces stay in the "up" state.
By default, the minimal number of required slave interfaces, which must stay in the "up" state in a bond of n
slave interfaces is n-1.
If one more slave interface fails (when n-2 slave interfaces stay in the "up" state), ClusterXL considers the
bond interface to be in the "down" state, even if the bond contains more than two slave interfaces.
If a smaller number of slave interfaces can pass the expected traffic, you can configure explicitly the
minimal number of required slave interfaces.
Divide your maximal expected traffic speed by the speed of your slave interfaces and round up the result to
find an applicable minimal number of required slave interfaces.

Notes :
n Cluster Members save the configuration in the $FWDIR/conf/cpha_bond_
ls_config.conf file.
n The commands below save the changes in this file.
n Each line in the file has this syntax:
<Name of Bond Interface> <Minimal Number of Required
Slave Interfaces>

In addition, see "Viewing Bond Interfaces" on page 218.

Syntax to add the minimal number of required slave interfaces for a specific Bond interface

Shell Command

Gaia N/A
Clish

Expert cphaconf bond_ls set <Name of Bond Interface> <Minimal Number

mode of Required Slave Interfaces>

ClusterXL R81 Administration Guide      |      193

 Configuring the Minimal Number of Required Slave Interfaces for Bond Load Sharing

Syntax to remove the configured minimal number of required slave interfaces for a specific Bond interface

Shell Command

Gaia Clish N/A

Expert mode cphaconf bond_ls remove <Name of Bond Interface>

Syntax to see the current configuration of the minimal number of required slave interfaces

Shell Command

Gaia Clish N/A

Expert mode cat $FWDIR/conf/cpha_bond_ls_config.conf

Procedure

Step Instructions

1 Connect to the command line on each Cluster Member.

2 Log in to the Expert mode.

3 Add or remove the minimal number of required slave interfaces for a specific Bond interface:
cphaconf bond_ls set <Bond> <Minimal Number of Slaves>

cphaconf bond_ls remove <Bond>

4 Examine the configuration:

cat $FWDIR/conf/cpha_bond_ls_config.conf

5 In SmartConsole, install the Access Control policy on this cluster object.

ClusterXL R81 Administration Guide      |      194

 Configuring the Minimal Number of Required Slave Interfaces for Bond Load Sharing

Example

[Expert@Member1:0]# cat $FWDIR/conf/cpha_bond_ls_config.conf

# ... (truncated for brevity) ...
# Example:
# bond0 2

[Expert@Member1:0]#

[Expert@Member1:0]# cphaconf bond_ls set bond1 2

Set operation succeeded

[Expert@Member1:0]# cat $FWDIR/conf/cpha_bond_ls_config.conf

# ... (truncated for brevity) ...
# Example:
# bond0 2

bond1 2
[Expert@Member1:0]#

[Expert@Member1:0]# cphaconf bond_ls remove bond1

Remove operation succeeded

[Expert@Member1:0]#

[Expert@Member1:0]# cat $FWDIR/conf/cpha_bond_ls_config.conf

# ... (truncated for brevity) ...
# Example:
# bond0 2

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      195

 Configuring the Multi-Version Cluster Mechanism

Configuring the Multi-Version Cluster

Mechanism
Description
This command lets you change the state of the Multi-Version Cluster (MVC) Mechanism - enable or
disable it.

Important:
n The MVC Mechanism is disabled by default.
n For limitations of the MVC Mechanism, see the R81 Installation and Upgrade
Guide > Chapter Upgrading Gateways and Clusters > Section Upgrading
ClusterXL, VSX Cluster, VRRP Cluster > Section Multi-Version Cluster
Upgrade.

Syntax

Shell Command

Gaia Clish set cluster member mvc {off | on}

Expert mode cphaconf mvc {off | on}

Parameters

Parameter Description

off Disables the MVC Mechanism on this Cluster Member.

on Enables the MVC Mechanism on this Cluster Member.

Notes:
n This command does not provide an output. To view the current state of the MVC
Mechanism, see "Viewing the State of the Multi-Version Cluster Mechanism" on
page 242.
n The change made with this command survives reboot.
n If a specific scenario requires you to disable the MVC Mechanism before the first
start of an R81 Cluster Member (for example, immediately after an upgrade to
R81), then disable it before the first policy installation on this Cluster Member.

ClusterXL R81 Administration Guide      |      196

 Monitoring and Troubleshooting Clusters

Monitoring and Troubleshooting

Clusters
This section describes how to monitor and troubleshooting clusters.

ClusterXL R81 Administration Guide      |      197

 ClusterXL Monitoring Commands

ClusterXL Monitoring Commands

Description
Use the monitoring commands to make sure that the cluster and the Cluster Members work properly, and
to define Critical Devices. A Critical Device (also known as a Problem Notification, or pnote) is a special
software device on each Cluster Member, through which the critical aspects for cluster operation are
monitored. When the critical monitored component on a Cluster Member fails to report its state on time, or
when its state is reported as problematic, the state of that member is immediately changed to 'Down'.

Syntax

Notes:
n In Gaia Clish:
Enter the show cluster<ESC><ESC> to see all the available commands.
n In Expert mode:
Run the cphaprob command see all the available commands.
You can run the cphaprob commands from Gaia Clish as well.
n Syntax legend:
1. Curly brackets or braces { }:
Enclose a list of available commands or parameters, separated by the
vertical bar |, from which user can enter only one.
2. Angle brackets < > :
Enclose a variable - a supported value user needs to specify explicitly.
3. Square brackets or brackets [ ]:
Enclose an optional command or parameter, which user can also enter.
n You can include these commands in scripts to run them automatically.
The meaning of each command is explained in the next sections.
Table: ClusterXL Monitoring Commands
Description Command in Command in
of Command Gaia Clish Expert Mode

Show states of Cluster Members and their names (see show cluster cphaprob [-
"Viewing Cluster State" on page 203) state vs <VSID>]
state

Show Critical Devices (Pnotes) and their states on the show cluster cphaprob [-
Cluster Member (see "Viewing Critical Devices" on members pnotes l] [-ia] [-
page 207) {all | problem} e] list

Show cluster interfaces on the cluster member (see show cluster cphaprob [-
"Viewing Cluster Interfaces" on page 214) members vs all] [-a]
interfaces {all [-m] if
| secured |
virtual |
vlans}

ClusterXL R81 Administration Guide      |      198

 ClusterXL Monitoring Commands

Table: ClusterXL Monitoring Commands (continued)

Description Command in Command in
of Command Gaia Clish Expert Mode

Show cluster bond configuration on the Cluster Member show cluster cphaprob
(see "Viewing Bond Interfaces" on page 218) bond {all | show_bond
name <bond_ [<bond_
name>} name>]

Show groups of bonds on the Cluster Member (see N / A cphaprob

"Viewing Bond Interfaces" on page 218) show_bond_
groups

Show (and reset) cluster failover statistics on the show cluster cphaprob [-
Cluster Member (see "Viewing Cluster Failover failover [reset reset {-c |
Statistics" on page 223) {count | -h}] [-l
history}] <count>]
show_
failover

Show information about the software version (including show cluster cphaprob
hotfixes) on the local Cluster Member and its release release
matches/mismatches with other Cluster Members (see
"Viewing Software Versions on Cluster Members" on
page 225)

Show Delta Sync statistics on the Cluster Member (see show cluster cphaprob [-
"Viewing Delta Synchronization" on page 226) statistics sync reset]
[reset] syncstat

Show Delta Sync statistics for the Connections table on show cluster cphaprob [-
the Cluster Member (see "Viewing Cluster Delta Sync statistics reset]
Statistics for Connections Table" on page 233) transport ldstat
[reset]

Show the Cluster Control Protocol (CCP) mode on the show cluster cphaprob [-
Cluster Member (see "Viewing Cluster Interfaces" on members vs all] -a
page 214) interfaces if
virtual

Show the IGMP membership of the Cluster Member show cluster cphaprob
(see "Viewing IGMP Status" on page 232) members igmp igmp

Show cluster unique IP's table on the Cluster Member show cluster cphaprob
(see "Viewing Cluster IP Addresses" on page 234) members ips tablestat

Show the Cluster Member ID Mode in local logs - by show cluster cphaprob
Member ID (default) or Member Name (see "Viewing members idmode names
the Cluster Member ID Mode in Local Logs" on
page 235)

ClusterXL R81 Administration Guide      |      199

 ClusterXL Monitoring Commands

Table: ClusterXL Monitoring Commands (continued)

Description Command in Command in
of Command Gaia Clish Expert Mode

Show interfaces, which the RouteD monitors on the show ospf cphaprob
Cluster Member when you configure OSPF (see interfaces routedifcs
"Viewing Interfaces Monitored by RouteD" on [detailed]
page 236)

Show roles of RouteD daemon on Cluster Members show cluster cphaprob

(see "Viewing Roles of RouteD Daemon on Cluster roles roles
Members" on page 237)

Show Cluster Correction Statistics (see "Viewing N / A cphaprob [{-

Cluster Correction Statistics" on page 238) d | -f | -
s}] corr

Show the Cluster Control Protocol (CCP) mode (see show cluster cphaprob -a
"Viewing the Cluster Control Protocol (CCP) Settings" members if
on page 240) interfaces
virtual

Show the Cluster Control Protocol (CCP) Encryption show cluster cphaprob
settings (see "Viewing the Cluster Control Protocol members ccpenc ccp_encrypt
(CCP) Settings" on page 240)

Shows the state of the Multi-Version Cluster (see show cluster N / A

"Viewing the State of the Multi-Version Cluster members mvc
Mechanism" on page 242)

Shows the latency and the drop rate of each interface N / A N / A

(see "Viewing Latency and Drop Rate of Interfaces" on
page 241)

Show Full Connectivity Upgrade statistics (see "Viewing N / A cphaprob

Full Connectivity Upgrade Statistics" on page 243) fcustat

ClusterXL R81 Administration Guide      |      200

 ClusterXL Monitoring Commands

List of the Gaia Clish show cluster commands

show cluster
      bond
            all
            name <Name of Bond>
      failover
      members
            ccpenc
            idmode
            igmp
            interfaces
                  all
                  secured
                  virtual
                  vlans
            ips
            mvc
            pnotes
                  all
                  problem
      release
      roles
      state
      statistics
            sync [reset]
            transport [reset]

ClusterXL R81 Administration Guide      |      201

 ClusterXL Monitoring Commands

List of the cphaprob commands

Note - Some commands are not applicable to 3rd party clusters.

cphaprob [-vs <VSID>] state

cphaprob [-reset {-c | -h}] [-l <count>] show_failover
cphaprob names
cphaprob [-reset] [-a] syncstat
cphaprob [-reset] ldstat
cphaprob [-l] [-i[a]] [-e] list
cphaprob [-vs all] [-a] [-m] if
cphaprob latency
cphaprob show_bond [<bond_name>]
cphaprob show_bond_groups
cphaprob igmp
cphaprob fcustat
cphaprob tablestat
cphaprob routedifcs
cphaprob roles
cphaprob release
cphaprob ccp_encrypt
cphaprob [{-d | -f | -s}] corr

ClusterXL R81 Administration Guide      |      202

 Viewing Cluster State

Viewing Cluster State

Description
This command lets you monitor the cluster status (after you set up the cluster).

Syntax

Shell Command

Gaia Clish 1. set virtual-system

<VSID>
2. show cluster state

Expert mode cphaprob [-vs <VSID>] state

Example

Member1> show cluster state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 (local) 11.22.33.245 100% ACTIVE(!) Member1

2 11.22.33.246 0% DOWN Member2

Active PNOTEs: COREXL

Last member state change event:

Event Code: CLUS-116505
State change: INIT -> ACTIVE(!)
Reason for state change: All other machines are dead (timeout), FULLSYNC PNOTE
Event time: Sun Sep 8 15:28:39 2019
v Cluster failover count:
Failover counter: 0
Time of counter reset: Sun Sep 8 15:28:21 2019 (reboot)

Member1>

Description of the "cphaprob state" command output fields:

Table: Description of the output fields
Field Description

Cluster Can be one of these:

Mode
n Load Sharing (Multicast).
n Load Sharing (Unicast).
n High Availability (Primary Up).
n High Availability (Active Up).
n Virtual System Load Sharing
n For third-party clustering products: Service, refer to Clustering Definitions and
Terms, for more information.

ClusterXL R81 Administration Guide      |      203

 Viewing Cluster State

Table: Description of the output fields (continued)

Field Description

ID n In the High Availability mode - indicates the Cluster Member priority, as

configured in the cluster object in SmartConsole.
n In Load Sharing mode - indicates the Cluster Member ID, as configured in the
cluster object in SmartConsole.

Unique Usually, shows the IP addresses of the Sync interfaces.

Address In some cases, can show IP addresses of other cluster interfaces.

Assigned n In the ClusterXL High Availability mode - shows the Active Cluster Member with
Load 100% load, and all other Standby Cluster Members with 0% load.
n In ClusterXL Load Sharing modes (Unicast and Multicast) - shows all Active
Cluster Members with 100% load.

State n In the ClusterXL High Availability mode, only one Cluster Member in a fully-
functioning cluster must be ACTIVE, and the other Cluster Members must be in
the STANDBY state.
n In the ClusterXL Load Sharing modes (Unicast and Multicast), all Cluster
Members in a fully-functioning cluster must be ACTIVE.
n In 3rd-party clustering configuration, all Cluster Members in a fully-functioning
cluster must be ACTIVE. This is because this command only reports the status
of the Full Synchronization process.
See the summary table below.

Name Shows the names of Cluster Members' objects as configured in SmartConsole.

Active Shows the Critical Devices that report theirs states as "problem" (see "Viewing
PNOTEs Critical Devices" on page 207).

Last member Shows information about the last time this Cluster Member changed its cluster state.
state change
event

Event Code Shows an event code.

For information, see sk125152.

State change Shows the previous cluster state and the new cluster state of this Cluster Member.

Reason for Shows the reason why this Cluster Member changed its cluster state.
state change

Event time Shows the date and the time when this Cluster Member changed its cluster state.

Last cluster Shows information about the last time a cluster failover occurred.
failover
event

Transition to Shows which Cluster Member became the new Active.

new ACTIVE

ClusterXL R81 Administration Guide      |      204

 Viewing Cluster State

Table: Description of the output fields (continued)

Field Description

Reason Shows the reason for the last cluster failover.

Event time Shows the date and the time of the last cluster failover.

Cluster Shows information about the cluster failovers.

failover
count

Failover Shows the number of cluster failovers since the boot.

counter Notes:
n This value survives reboot.
n This counter is synchronized between Cluster Members.

Time of Shows the date and the time of the last counter reset, and the reset initiator.
counter reset

When you examine the state of the Cluster Member, consider whether it forwards packets, and whether it
has a problem that prevents it from forwarding packets. Each state reflects the result of a test on critical
devices. This table shows the possible cluster states, and whether or not they represent a problem.
Table: Description of the cluster states
Is this
Cluster Forwarding
Description state a
State packets?
problem?

ACTIVE Everything is OK. Yes No

ACTIVE(!) A problem was detected, but the Cluster Member still Yes Yes
ACTIVE forwards packets, because it is the only member in the
(!F) cluster, or because there are no other Active members in the
ACTIVE cluster. In any other situation, the state of the member is
(!P) Down.
ACTIVE n ACTIVE(!) - See above.
(!FP) n ACTIVE(!F) - See above. Cluster Member is in the
freeze state.
n ACTIVE(!P) - See above. This is the Pivot Cluster
Member in Load Sharing Unicast mode.
n ACTIVE(!FP) - See above. This is the Pivot Cluster
Member in Load Sharing Unicast mode and it is in the
freeze state.

DOWN One of the Critical Devices reports its state as "problem" No Yes
(see "Viewing Critical Devices" on page 207).

LOST The peer Cluster Member lost connectivity to this local No Yes
Cluster Member (for example, while the peer Cluster Member
is rebooted).

ClusterXL R81 Administration Guide      |      205

 Viewing Cluster State

Table: Description of the cluster states (continued)

Is this
Cluster Forwarding
Description state a
State packets?
problem?

READY State Ready means that the Cluster Member recognizes itself No No
as a part of the cluster and is literally ready to go into action,
but, by design, something prevents it from taking action.
Possible reasons that the Cluster Member is not yet Active
include:
n Not all required software components were loaded and
initialized yet and/or not all configuration steps finished
successfully yet. Before a Cluster Member becomes
Active, it sends a message to the rest of the Cluster
Members, to check if it can become Active. In High
Availability mode it checks if there is already an Active
member and in Load Sharing Unicast mode it checks if
there is a Pivot member already. The member remains
in the Ready state until it receives the response from
the rest of the Cluster Members and decides which,
which state to choose next (Active, Standby, Pivot, or
non-Pivot).
n Software installed on this Cluster Member has a higher
version than all the other Cluster Members. For
example, when a cluster is upgraded from one version
of Check Point Security Gateway to another, and the
Cluster Members have different versions of Check
Point Security Gateway, the Cluster Members with the
new version have the Ready state, and the Cluster
Members with the previous version have the
Active/Active Attention state.
This applies only when the Multi-Version Cluster
Mechanism is disabled (see "Viewing the State of the
Multi-Version Cluster Mechanism" on page 242).
See sk42096 for a solution.

STANDBY Applies only to a High Availability mode. Means that the No No

Cluster Member waits for an Active Cluster Member to fail in
order to start packet forwarding.

BACKUP Applies only to a VSX Cluster in Virtual System Load Sharing No No

mode with three or more Cluster Members configured.
State of a Virtual System on a third (and so on) VSX Cluster
Member.

INIT The Cluster Member is in the phase after the boot and until No No
the Full Sync completes.

ClusterXL R81 Administration Guide      |      206

 Viewing Critical Devices

Viewing Critical Devices

Description
There are a number of built-in Critical Devices, and the Administrator can define additional Critical Devices.
When a Critical Device reports its state as a "problem", the Cluster Member reports its state as "DOWN".
To see the list of Critical Devices on a Cluster Member, and of all the other Cluster Members, run the
commands listed below on the Cluster Member.
Table: Built-in Critical Devices
Meaning of the Meaning of the
Critical Device Description
"OK" state "problem" state

Problem Monitors all the Critical Devices. None of the At least one of the
Notification Critical Devices Critical Devices on this
on this Cluster Cluster Member
Member report its reports its state as
state as problem.
problem.

Init Monitors if "HA module" was This Cluster

initialized successfully. See Member receives
sk36372. cluster state
information from
peer Cluster
Members.

Interface Monitors the state of cluster All cluster At least one of the
Active Check interfaces. interfaces on this cluster interfaces on
Cluster Member this Cluster Member is
are up (CCP down (CCP packets
packets are sent are not sent and/or
and received on received on time).
all cluster
interfaces).

Load Pnote is currently not used (see

Balancing sk36373).
Configuration

Recovery Monitors the state of a Virtual State of a Virtual State of a Virtual

Delay System (see sk92353). System can be System cannot be
changed on this changed yet on this
Cluster Member. Cluster Member.

ClusterXL R81 Administration Guide      |      207

 Viewing Critical Devices

Table: Built-in Critical Devices (continued)

Meaning of the Meaning of the
Critical Device Description
"OK" state "problem" state

CoreXL Monitors CoreXL configuration for Number of Number of configured

Configuration inconsistencies on all Cluster configured CoreXL Firewall
Members. CoreXL Firewall instances on this
instances on this Cluster Member is
Cluster Member different from peer
is the same as on Cluster Members.
all peer Cluster Important - A Cluster
Members. Member with a
greater number of
CoreXL Firewall
instances changes its
state to DOWN.

Fullsync Monitors if Full Sync on this This Cluster This Cluster Member
Cluster Member completed Member was not able to
successfully. completed Full complete Full Sync.
Sync
successfully.

Policy Monitors if the Security Policy is This Cluster Security Policy is not
installed. Member currently installed on
successfully this Cluster Member.
installed Security
Policy.

fwd Monitors the Security Gateway fwd daemon on fwd daemon on this
process called fwd. this Cluster Cluster Member did
Member reported not report its state on
its state on time. time.

cphad Monitors the ClusterXL process cphamcset cphamcset daemon

called cphamcset. daemon on this on this Cluster
also see the Cluster Member Member did not report
$FWDIR/log/cphamcset.elg reported its state its state on time.
file. on time.

routed Monitors the Gaia process called routed daemon routed daemon on
routed. on this Cluster this Cluster Member
Member reported did not report its state
its state on time. on time.

cvpnd Monitors the Mobile Access back- cvpnd daemon cvpnd daemon on
end process called cvpnd. on this Cluster this Cluster Member
This pnote appears if Mobile Member reported did not report its state
Access Software Blade is enabled. its state on time. on time.

ted Monitors the Threat Emulation ted daemon on ted daemon on this
process called ted. this Cluster Cluster Member did
Member reported not report its state on
its state on time. time.

ClusterXL R81 Administration Guide      |      208

 Viewing Critical Devices

Table: Built-in Critical Devices (continued)

Meaning of the Meaning of the
Critical Device Description
"OK" state "problem" state

VSX Monitors all Virtual Systems in VSX On VS0, means Minimum of blocking
Cluster. that states of all states of all Virtual
Virtual Systems Systems is not "active"
are not Down on (the VSIDs will be
this Cluster printed on the line
Member. Problematic
On other Virtual VSIDs:) on this
Systems, means Cluster Member.
that VS0 is alive
on this Cluster
Member.

Instances This pnote appears in VSX HA The number of There is a mismatch

mode (not VSLS) cluster. CoreXL Firewall between the number
instances in the of CoreXL Firewall
received CCP instances in the
packet matches received CCP packet
the number of and the number of
loaded CoreXL loaded CoreXL
Firewall instances Firewall instances on
on this VSX this VSX Cluster
Cluster Member Member or this Virtual
or this Virtual System (see
System. sk106912).

Hibernating This pnote appears in VSX VSLS This Virtual

mode cluster with 3 and more System is in
Cluster Members. This pnote "Backup"
shows if this Virtual System is in (hibernated) state
"Backup" (hibernated) state. Also on this Cluster
see sk114557. Member.

admin_down Monitors the Critical Device User ran the

admin_down. clusterXL_admin
down command on
this Cluster Member.
See "The clusterXL_
admin Script" on
page 273.

host_monitor Monitors the Critical Device All monitored IP At least one of the
host_monitor. addresses on this monitored IP
User executed the Cluster Member addresses on this
$FWDIR/bin/clusterXL_ replied to pings. Cluster Member did
monitor_ips script. not reply to at least
See "The clusterXL_monitor_ips one ping.
Script" on page 277.

ClusterXL R81 Administration Guide      |      209

 Viewing Critical Devices

Table: Built-in Critical Devices (continued)

Meaning of the Meaning of the
Critical Device Description
"OK" state "problem" state

A name of a user User executed the All monitored At least one of the
space process $FWDIR/bin/clusterXL_ user space monitored user space
(except fwd, monitor_process script. processes on this on this Cluster
routed, cvpnd, See "The clusterXL_monitor_ Cluster Member Member processes is
ted) process Script" on page 281. are running. not running.

Syntax

Shell Command

Gaia Clish show cluster members pnotes {all | problem}

Expert mode cphaprob [-l] [-ia] [-e] list

Where:

Command Description

show cluster Shows cluster full list of Critical Devices

members pnotes
all

show cluster Prints the list of all the "Built-in Devices" and the "Registered
members pnotes Devices"
problem

cphaprob -l Prints the list of all the "Built-in Devices" and the "Registered
Devices"

cphaprob -i list When there are no issues on the Cluster Member, shows:
There are no pnotes in problem state
When a Critical Device reports a problem, prints only the Critical Device that
reports its state as "problem".

cphaprob -ia When there are no issues on the Cluster Member, shows:
list There are no pnotes in problem state
When a Critical Device reports a problem, prints the Critical Device
"Problem Notification" and the Critical Device that reports its state
as "problem"

cphaprob -e list When there are no issues on the Cluster Member, shows:
There are no pnotes in problem state
When a Critical Device reports a problem, prints only the Critical Device that
reports its state as "problem"

ClusterXL R81 Administration Guide      |      210

 Viewing Critical Devices

Related topics
n "Reporting the State of a Critical Device" on page 184
n "Registering a Critical Device" on page 181
n "Registering Critical Devices Listed in a File" on page 185
n "Unregistering a Critical Device" on page 183
n "Unregistering All Critical Devices" on page 187

ClusterXL R81 Administration Guide      |      211

 Viewing Critical Devices

Examples

Example 1 - Critical Device 'fwd'

Critical Device fwd reports its state as problem because the fwd process is down.

[Expert@Member1:0]# cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Current state: OK

Device Name: Recovery Delay

Current state: OK

Device Name: CoreXL Configuration

Current state: OK

Registered Devices:

Device Name: Fullsync

Registration number: 0
Timeout: none
Current state: OK
Time since last report: 1753.7 sec

Device Name: Policy

Registration number: 1
Timeout: none
Current state: OK
Time since last report: 1753.7 sec

Device Name: routed

Registration number: 2
Timeout: none
Current state: OK
Time since last report: 940.3 sec

Device Name: fwd

Registration number: 3
Timeout: 30 sec
Current state: problem
Time since last report: 1782.9 sec
Process Status: DOWN

Device Name: cphad

Registration number: 4
Timeout: 30 sec
Current state: OK
Time since last report: 1778.3 sec
Process Status: UP

Device Name: VSX

Registration number: 5
Timeout: none
Current state: OK
Time since last report: 1773.3 sec

Device Name: Init

Registration number: 6
Timeout: none
Current state: OK
Time since last report: 1773.3 sec

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      212

 Viewing Critical Devices

Example 2 - Critical Device 'CoreXL Configuration'

Critical Device CoreXL Configuration reports its state as problem because the numbers of
CoreXL Firewall instances do not match between the Cluster Members.

[Expert@Member1:0]# cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Current state: OK

Device Name: Recovery Delay

Current state: OK

Device Name: CoreXL Configuration

Current state: problem (non-blocking)

Registered Devices:

Device Name: Fullsync

Registration number: 0
Timeout: none
Current state: OK
Time since last report: 1753.7 sec

Device Name: Policy

Registration number: 1
Timeout: none
Current state: OK
Time since last report: 1753.7 sec

Device Name: routed

Registration number: 2
Timeout: none
Current state: OK
Time since last report: 940.3 sec

Device Name: fwd

Registration number: 3
Timeout: 30 sec
Current state: OK
Time since last report: 1782.9 sec
Process Status: UP

Device Name: cphad

Registration number: 4
Timeout: 30 sec
Current state: OK
Time since last report: 1778.3 sec
Process Status: UP

Device Name: VSX

Registration number: 5
Timeout: none
Current state: OK
Time since last report: 1773.3 sec

Device Name: Init

Registration number: 6
Timeout: none
Current state: OK
Time since last report: 1773.3 sec

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      213

 Viewing Cluster Interfaces

Viewing Cluster Interfaces

Description
This command lets you see the state of the Cluster Member interfaces and the virtual cluster interfaces.
ClusterXL treats the interfaces as Critical Devices. ClusterXL makes sure that interfaces can send and
receive CCP packets.
ClusterXL also sets the required minimal number of functional interfaces to the largest number of
functional interfaces ClusterXL detected since the last reboot. If the number of functional interfaces is less
than the required number, ClusterXL declares the Cluster Member as failed and starts a failover. The
same applies to the synchronization interfaces, where only good synchronization interfaces are counted.
When an interface is DOWN, it means that the interface cannot receive or send CCP packets, or both. An
interface may also be able to receive, but not send CCP packets. The time you see in the command's
output is the number of seconds that elapsed since the interface was last able to receive or send a CCP
packet.

Syntax

Shell Command

Gaia Clish 1. set virtual-system <VSID>

2. show cluster members interfaces {all | secured | virtual
| vlans}

Expert cphaprob [-vs all] [-a] [-m] if

mode

ClusterXL R81 Administration Guide      |      214

 Viewing Cluster Interfaces

Where:

Command Description

show cluster members interfaces Shows full list of all cluster interfaces:
all
n including the number of required interfaces
n including Network Objective
n including VLAN monitoring mode, or list of
monitored VLAN interfaces

show cluster members interfaces Shows only cluster interfaces (Cluster and Sync) and
secured their states:
n without Network Objective
n without VLAN monitoring mode
n without monitored VLAN interfaces

show cluster members interfaces Shows full list of cluster virtual interfaces and their
virtual states:
n including the number of required interfaces
n including Network Objective
n without VLAN monitoring mode
n without monitored VLAN interfaces

show cluster members interfaces Shows only monitored VLAN interfaces

vlans

cphaprob if Shows only cluster interfaces (Cluster and Sync) and

their states:

n without Network Objective

n without VLAN monitoring mode
n without monitored VLAN interfaces

cphaprob -a if Shows full list of cluster interfaces and their states:

n including the number of required interfaces
n including Network Objective
n without VLAN monitoring mode
n without monitored VLAN interfaces

cphaprob -a -m if Shows full list of all cluster interfaces and their states:
n including the number of required interfaces
n including Network Objective
n including VLAN monitoring mode, or list of
monitored VLAN interfaces

Output
The output of these commands must be identical to the configuration in the cluster object's Network
Management page in SmartConsole.

ClusterXL R81 Administration Guide      |      215

 Viewing Cluster Interfaces

Example

[Expert@Member1:0]# cphaprob -a -m if

CCP mode: Manual (Unicast)

Required interfaces: 4
Required secured interfaces: 1

Interface Name: Status:

eth0 UP
eth1 (S) UP
eth2 (LM) UP
bond1 (LS) UP

S - sync, LM - link monitor, HA/LS - bond type

Virtual cluster interfaces: 3

eth0 192.168.3.247
eth2 44.55.66.247
bond1 77.88.99.247

No VLANs are monitored on the member

[Expert@Member1:0]#

Description of the "cphaprob -a -m if" command output fields:

Table: Description of the output fields
Field, or Text Description

CCP mode Shows the CCP mode.

The default mode is Unicast.

Important - In R81, the CCP always runs in the unicast

mode.

Required interfaces Shows the total number of monitored cluster interfaces, including
the Sync interface.
This number is based on the configuration of the cluster object >
Network Management page.

Required secured interfaces Shows the total number of the required Sync interfaces.
This number is based on the configuration of the cluster object >
Network Management page.

Non-Monitored This means that Cluster Member does not monitor the state of this
interface.
In SmartConsole, in the cluster object > Network Management
page, administrator configured the Network Type Private for this
interface.

ClusterXL R81 Administration Guide      |      216

 Viewing Cluster Interfaces

Table: Description of the output fields (continued)

Field, or Text Description

UP This means that Cluster Member monitors the state of this interface.
The current cluster state of this interface is UP, which means this
interface can send and receive CCP packets.
In SmartConsole, in the cluster object > Network Management
page, administrator configured one of these Network Types for this
interface: Cluster, Sync , or Cluster + Sync .

DOWN This means that Cluster Members monitors the state of this
interface.
The current cluster state of this interface is DOWN, which means
this interface cannot send CCP packets, receive CCP packets, or
both.
In SmartConsole, in the cluster object > Network Management
page, administrator configured one of these Network Types for this
interface: Cluster, Sync , or Cluster + Sync .

(S) This interface is a Sync interface.

In SmartConsole, in the cluster object > Network Management
page, administrator configured one of these Network Types for this
interface: Sync , or Cluster + Sync .

(LM) This interface is configured in the $FWDIR/conf/cpha_link_

monitoring.conf file.
Cluster Member monitors only the link on this interface (does not
monitor the received or sent CCP packets).
See "Configuring Link Monitoring on the Cluster Interfaces" on
page 107.

(HA) This interface is a Bond interface in High Availability mode.

(LS) This interface is a Bond interface in Load Sharing mode.

Virtual cluster interfaces Shows the total number of the configured virtual cluster interfaces.
This number is based on the configuration of the cluster object >
Network Management page.

No VLANs are monitored on Shows the VLAN monitoring mode - there are no VLAN interfaces
the member configured on the cluster interfaces.

Monitoring mode is Monitor Shows the VLAN monitoring mode - there are some VLAN interfaces
all VLANs: All VLANs are configured on the cluster interfaces, and Cluster Member monitors
monitored all VLAN IDs.

Monitoring mode is Monitor Shows the VLAN monitoring mode - there are some VLAN interfaces
specific VLAN: Only specified configured on the cluster interfaces, and Cluster Member monitors
VLANs are monitored only specific VLAN IDs.

ClusterXL R81 Administration Guide      |      217

 Viewing Bond Interfaces

Viewing Bond Interfaces

Description
This command lets you see the configuration of bond interfaces and their slave interfaces.

Syntax

Shell Command

Gaia Clish 1. show cluster bond {all | name <bond_

name>}
2. show bonding groups

Expert mode cphaprob show_bond [<bond_name>]

cphaprob show_bond_groups

Where:

Command Description

show cluster bond all Shows configuration of all configured bond interfaces
show bonding groups
cphaprob show_bond

show cluster bond name <bond_ Shows configuration of the specified bond interface
name>
cphaprob show_bond <bond_name>

cphaprob show_bond_groups Shows the configured Groups of Bonds and their

settings.

ClusterXL R81 Administration Guide      |      218

 Viewing Bond Interfaces

Examples

Example 1 - 'cphaprob show_bond'

[Expert@Member2:0]# cphaprob show_bond

|Slaves |Slaves |Slaves

Bond name |Mode |State |configured |link up |required
-----------+-------------------+------+-----------+--------+--------
bond1 | High Availability | UP | 2 | 2 | 1

Legend:
-------
UP! - Bond interface state is UP, yet attention is required
Slaves configured - number of slave interfaces configured on the bond
Slaves link up - number of operational slaves
Slaves required - minimal number of operational slaves required for bond to be UP

[Expert@Member2:0]#

Member2> show bonding groups

Bonding Interface: 1
Bond Configuration
xmit-hash-policy Not configured
down-delay 200
primary Not configured
lacp-rate Not configured
mode active-backup
up-delay 200
mii-interval 100
Bond Interfaces
eth3
eth4
Member2>

Description of the output fields for the "cphaprob show_bond" and "show cluster bond all"
commands:
Table: Description of the output fields
Field Description

Bond name Name of the Gaia bonding group.

Mode Bonding mode of this Gaia bonding group.

One of these:
n High Availability
n Load Sharing

State State of the Gaia bonding group:

n UP - Bond interface is fully operational
n UP! - Bond interface state is UP, yet attention is required
n DOWN - Bond interface failed

Slaves Total number of physical slave interfaces configured in this Gaia bonding group.
configured

Slaves link Number of operational physical slave interfaces in this Gaia bonding group.
up

ClusterXL R81 Administration Guide      |      219

 Viewing Bond Interfaces

Table: Description of the output fields (continued)

Field Description

Slaves Minimal number of operational physical slave interfaces required for the state of this
required Gaia bonding group to be UP.

Example 2 - 'cphaprob show_bond <bond_name>'

[Expert@Member2:0]# cphaprob show_bond bond1

Bond name: bond1

Bond mode: High Availability
Bond status: UP

Configured slave interfaces: 2

In use slave interfaces: 2
Required slave interfaces: 1

Slave name | Status | Link

----------------+-----------------+-------
eth4 | Active | Yes
eth3 | Backup | Yes

[Expert@Member2:0]#

Description of the output fields for the "cphaprob show_bond <bond_name>" and "show
cluster bond name <bond_name>" commands:
Table: Description of the output fields
Field Description

Bond Name of the Gaia bonding group.

name

Bond Bonding mode of this Gaia bonding group. One of these:

mode
n High Availability
n Load Sharing

Bond Status of the Gaia bonding group. One of these:

status
n UP - Bond interface is fully operational
n UP! - Bond interface state is UP, yet attention is required
n DOWN - Bond interface failed

Configured Total number of physical slave interfaces configured in this Gaia bonding group.
slave
interfaces

In use Number of operational physical slave interfaces in this Gaia bonding group.
slave
interfaces

Required Minimal number of operational physical slave interfaces required for the state of this
slave Gaia bonding group to be UP.
interfaces

ClusterXL R81 Administration Guide      |      220

 Viewing Bond Interfaces

Table: Description of the output fields (continued)

Field Description

Slave Names of physical slave interfaces configured in this Gaia bonding group.
name

Status Status of physical slave interfaces in this Gaia bonding group.

One of these:
n Active - In High Availability or Load Sharing bonding mode. This slave
interface is currently handling traffic.
n Backup - In High Availability bonding mode only. This slave interface is ready
and can support internal bond failover.
n Not Available - In High Availability or Load Sharing bonding mode. The
physical link on this slave interface is lost, or this Cluster Member is in status
Down. The bond cannot failover internally in this state.

Link State of the physical link on the physical slave interfaces in this Gaia bonding group.
One of these:
n Yes - Link is present
n No - Link is lost

Example 3 - 'cphaprob show_bond_groups'

[Expert@Member2:0]# cphaprob show_bond_groups

| Required | Bonds | Bonds

Group of bonds name | State | active bonds | in group | status
--------------------+-----------+--------------+----------+--------+
GoB0 | UP | 1 | |
| | | bond1 | UP
| | | bond2 | UP

Legend:
---------
Bonds in group - a list of the bonds in the bond group
Required active bonds - number of required active bonds
[Expert@Member2:0]#

Description of the output fields for the "cphaprob show_bond_groups" command:

Table: Description of the output fields
Field Description

Group of bonds name Name of the Group of Bonds.

State State of the Group of Bonds. One of these:

n UP - Group of Bonds is fully operational

n DOWN - Group of Bonds failed

Required active bonds Number of required active bonds in this Group of Bonds.

Bonds in group Names of the Gaia bond interfaces configured in this Group of Bonds.

ClusterXL R81 Administration Guide      |      221

 Viewing Bond Interfaces

Table: Description of the output fields (continued)

Field Description

Bonds status State of the Gaia bond interface. One of these:

n UP - Bond interface is fully operational
n DOWN - Bond interface failed

ClusterXL R81 Administration Guide      |      222

 Viewing Cluster Failover Statistics

Viewing Cluster Failover Statistics

Description
This command lets you see the cluster failover statistics on the Cluster Member:
n Number of failovers that happened
n Failover reason
n The time of the last failover event

Syntax to show the statistics

Shell Command

Gaia Clish show cluster failover

Expert mode cphaprob [-l <number>] show_failover

Syntax to reset the statistics

Shell Command

Gaia Clish show cluster failover reset {count | history}

Expert mode cphaprob -reset {-c | -h} show_failover

Parameters

Parameter Description

-l <number> Specifies how many of last failover events to show (between 1 and 50)

count Resets the counter of failover events

-c

history Resets the history of failover events

-h

ClusterXL R81 Administration Guide      |      223

 Viewing Cluster Failover Statistics

Example

[Expert@Member1:0]# cphaprob show_failover

Last cluster failover event:

Transition to new ACTIVE: Member 2 -> Member 1
Reason: ADMIN_DOWN PNOTE
Event time: Sun Sep 8 18:21:44 2019

Cluster failover count:

Failover counter: 1
Time of counter reset: Sun Sep 8 16:08:34 2019 (reboot)

Cluster failover history (last 20 failovers since reboot/reset on Sun Sep 8 16:08:34 2019):

No. Time: Transition: CPU: Reason:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - -
1 Sun Sep 8 18:21:44 2019 Member 2 -> Member 1 01 ADMIN_DOWN PNOTE

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      224

 Viewing Software Versions on Cluster Members

Viewing Software Versions on Cluster Members

Description
This command lets you see information about the software version (including private hotfixes) on the local
Cluster Member and its matches / mismatches with other Cluster Members.

Syntax

Shell Command

Gaia Clish show cluster release

Expert mode cphaprob release

Example

[Expert@Member1:0]# cphaprob release

Release: R80.40 T136

Kernel build: 994000117

FW1 build: 994000116
FW1 private fixes: None

ID SW release

1 (local) R80.40 T136

2 R80.40 T136

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      225

 Viewing Delta Synchronization

Viewing Delta Synchronization

Heavily loaded clusters and clusters with geographically separated members pose special challenges.
High connection rates, and large distances between the members can lead to delays that affect the
operation of the cluster.
Monitor the operation of the State Synchronization mechanism in highly loaded and distributed clusters.

Perform these troubleshooting steps:

1. Examine the Delta Sync statistics counters:

Shell Command

Gaia Clish show cluster statistics sync

Expert mode cphaprob syncstat

2. Change the values of the applicable synchronization global configuration parameters.

3. Reset the Delta Sync statistics counters:

Shell Command

Gaia Clish show cluster statistics sync reset

Expert mode cphaprob -reset syncstat

4. Examine the Delta Sync statistics to see if the problem is solved.

5. Solve any identified problem.

ClusterXL R81 Administration Guide      |      226

 Viewing Delta Synchronization

Example output of the "show cluster statistics sync" and "cphaprob syncstat"
commands from a Cluster Member:

Delta Sync Statistics

Sync status: OK

Drops:
Lost updates................................. 0
Lost bulk update events...................... 0
Oversized updates not sent................... 0

Sync at risk:
Sent reject notifications.................... 0
Received reject notifications................ 0

Sent messages:
Total generated sync messages................ 26079
Sent retransmission requests................. 0
Sent retransmission updates.................. 0
Peak fragments per update.................... 1

Received messages:
Total received updates....................... 3710
Received retransmission requests............. 0

Sync Interface:
Name......................................... eth1
Link speed................................... 1000Mb/s
Rate......................................... 46000 [Bps]
Peak rate.................................... 46000 [Bps]
Link usage................................... 0%
Total........................................ 376827[KB]

Queue sizes (num of updates):

Sending queue size........................... 512
Receiving queue size......................... 256
Fragments queue size......................... 50

Timers:
Delta Sync interval (ms)..................... 100

Reset on Sun Sep 8 16:09:15 2019 (triggered by fullsync).

Each section of the output is described below.

The "Sync status:" section

This section shows the status of the Delta Sync mechanism. One of these:
n Sync status: OK
n Sync status: Off - Full-sync failure
n Sync status: Off - Policy installation failure
n Sync status: Off - Cluster module not started
n Sync status: Off - SIC failure
n Sync status: Off - Full-sync checksum error
n Sync status: Off - Full-sync received queue is full
n Sync status: Off - Release version mismatch

ClusterXL R81 Administration Guide      |      227

 Viewing Delta Synchronization

n Sync status: Off - Connection to remote member timed-out

n Sync status: Off - Connection terminated by remote member
n Sync status: Off - Could not start a connection to remote member
n Sync status: Off - cpstart
n Sync status: Off - cpstop
n Sync status: Off - Manually disabled sync
n Sync status: Off - Was not able to start for more than X second
n Sync status: Off - Boot
n Sync status: Off - Connectivity Upgrade (CU)
n Sync status: Off - cphastop
n Sync status: Off - Policy unloaded
n Sync status: Off - Hibernation
n Sync status: Off - OSU deactivated
n Sync status: Off - Sync interface down
n Sync status: Fullsync in progress
n Sync status: Problem (Able to send sync packets, unable to receive
sync packets)
n Sync status: Problem (Able to send sync packets, saving incoming
sync packets)
n Sync status: Problem (Able to send sync packets, able to receive
sync packets)
n Sync status: Problem (Unable to send sync packets, unable to
receive sync packets)
n Sync status: Problem (Unable to send sync packets, saving incoming
sync packets)
n Sync status: Problem (Unable to send sync packets, able to receive
sync packets)

The "Drops:" section

This section shows statistics for drops on the Delta Sync network.

ClusterXL R81 Administration Guide      |      228

 Viewing Delta Synchronization

Table: Description of the output fields

Field Description

Lost Shows how many Delta Sync updates this Cluster Member considers as lost (based on
updates sequence numbers in CCP packets).
If this counter shows a value greater than 0, this Cluster Member lost Delta Sync
updates.
Possible mitigation:
Increase the size of the Sending Queue and the size of the Receiving Queue:
n Increase the size of the Sending Queue, if the counter Received reject
notification is increasing.
n Increase the size of the Receiving Queue, if the counter Received reject
notification is not increasing.

Lost bulk Shows how many times this Cluster Member missed Delta Sync updates.
update (bulk update = twice the size of the local receiving queue)
events This counter increases when this Cluster Member receives a Delta Sync update with a
sequence number much greater than expected. This probably indicates some
networking issues that cause massive packet drops.
This counter increases when the amount of missed Delta Sync updates is more than
twice the local Receiving Queue Size.
Possible mitigation:
n If the counter's value is steady, this might indicate a one-time synchronization
problem that can be resolved by running manual Full Sync. See sk37029.
n If the counter's value keeps increasing, probable there are some networking
issues. Increase the sizes of both the Receiving Queue and Sending Queue.

Oversized Shows how many oversized Delta Sync updates were discarded before sending them.
updates This counter increases when Delta Sync update is larger than the local Fragments
not sent Queue Size.
Possible mitigation:
n If the counter's value is steady, increase the size of the Sending Queue.
n If the counter's value keeps increasing, contact Check Point Support.

The "Sync at risk:" section

This section shows statistics that the Sending Queue is at full capacity and rejects Delta Sync
retransmission requests.
Table: Description of the output fields
Field Description

Sent reject Shows how many times this Cluster Member rejected Delta Sync retransmission
notifications requests from its peer Cluster Members, because this Cluster Member does not hold
the requested Delta Sync update anymore.

Received Shows how many reject notifications this Cluster Member received from its peer
reject Cluster Members.
notification

ClusterXL R81 Administration Guide      |      229

 Viewing Delta Synchronization

The "Sent updates:" section

This section shows statistics for Delta Sync updates sent by this Cluster Member to its peer Cluster
Members.
Table: Description of the output fields
Field Description

Total Shows how many Delta Sync updates were generated.

generated This counts the Delta Sync updates, Retransmission Requests, Retransmission
sync Acknowledgments, and so on.
messages

Sent Shows how many times this Cluster Member asked its peer Cluster Members to
retransmission retransmit specific Delta Sync update(s).
requests Retransmission requests are sent when certain Delta Sync updates (with a
specified sequence number) are missing, while the sending Cluster Member
already received Delta Sync updates with advanced sequences.
Note - Compare the number of Sent retransmission requests to the Total
generated sync messages of the other Cluster Members.
A large counter's value can imply connectivity problems. If the counter's value is
unreasonably high (more than 30% of the Total generated sync messages of
other Cluster Members), contact Check Point Support equipped with the entire
output and a detailed description of the network topology and configuration.

Sent Shows how many times this Cluster Member retransmitted specific Delta Sync
retransmission update(s) at the requests from its peer Cluster Members.
updates

Peak Shows the peak amount of fragments in the Fragments Queue on this Cluster
fragments per Member (usually, should be 1).
update

The "Received updates:" section

This section shows statistics for Delta Sync updates that were received by this Cluster Member from its
peer Cluster Members.
Table: Description of the output fields
Field Description

Total received Shows the total number of Delta Sync updates this Cluster Member received from
updates its peer Cluster Members.
This counts only Delta Sync updates (not Retransmission Requests,
Retransmission Acknowledgments, and others).

Received Shows how many retransmission requests this Cluster Member received from its
retransmission peer Cluster Members.
requests A large counter's value can imply connectivity problems. If the counter's value is
unreasonably high (more than 30% of the Total generated sync messages on
this Cluster Member), contact Check Point Support equipped with the entire
output and a detailed description of the network topology and configuration.

ClusterXL R81 Administration Guide      |      230

 Viewing Delta Synchronization

The "Queue sizes (num of updates):" section

This section shows the sizes of the Delta Sync queues.

Table: Description of the output fields
Field Description

Sending Shows the size of the cyclic queue, which buffers all the Delta Sync updates that were
queue already sent until it receives an acknowledgment from the peer Cluster Members.
size This queue is needed for retransmitting the requested Delta Sync updates.
Each Cluster Member has one Sending Queue.
Default: 512 Delta Sync updates, which is also the minimal value.

Receiving Shows the size of the cyclic queue, which buffers the received Delta Sync updates in
queue two cases:
size
n When Delta Sync updates are missing, this queue is used to hold the remaining
received Delta Sync updates until the lost Delta Sync updates are retransmitted
(Cluster Members must keep the order, in which they save the Delta Sync
updates in the kernel tables).
n This queue is used to re-assemble a fragmented Delta Sync update.
Each Cluster Member has one Receiving Queue.
Default: 256 Delta Sync updates, which is also the minimal value.

Fragments Shows the size of the queue, which is used to prepare a Delta Sync update before
queue moving it to the Sending Queue.
size Notes:
n This queue must be smaller than the Sending Queue.
n This queue must be significantly smaller than the Receiving Queue.
Default: 50 Delta Sync updates, which is also the minimal value.

The "Timers:" section

This section shows the Delta Sync timers.

Field Description

Delta Sync Shows the interval at which this Cluster Member sends the Delta Sync updates
interval (ms) from its Sending Queue.
The base time unit is 100ms (or 1 tick).
Default: 100 ms, which is also the minimum value.
See Increasing the Sync Timer.

The "Reset on XXX (triggered XXX)" section

Shows the date and the time of last statistics reset.

In parentheses, it shows how the last statistics was triggered - "manually", or "by fullsync".

ClusterXL R81 Administration Guide      |      231

 Viewing IGMP Status

Viewing IGMP Status

Description
This command lets you view the IGMP membership status.

Syntax

Shell Command

Gaia Clish show cluster members igmp

Expert mode cphaprob igmp

Example

[Expert@Member1:0]# cphaprob igmp

IGMP Membership: Enabled

Supported Version: 2
Report Interval [sec]: 60

IGMP queries are replied only by Operating System

Interface Host Group Multicast Address Last ver. Last Query[sec]

------------------------------------------------------------------------------
eth0 224.168.3.247 01:00:5e:28:03:f7 N/A N/A
eth1 224.22.33.250 01:00:5e:16:21:fa N/A N/A
eth2 224.55.66.247 01:00:5e:37:42:f7 N/A N/A

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      232

 Viewing Cluster Delta Sync Statistics for Connections Table

Viewing Cluster Delta Sync Statistics for Connections Table

Description
This command lets you see Delta Sync statistics about the operations performed in the Connections
Kernel Table (id 8158).
The output shows operations such as creating a new connection (SET), updating a connection
(REFRESH), deleting a connection (DELETE), and so on.

Syntax

Shell Command

Gaia Clish show cluster statistics transport [reset]

Expert mode cphaprob [-reset] ldstat

The "reset" flag resets the kernel statistics, which were collected since the last reboot or reset.

Example

[Expert@Member1:0]# cphaprob ldstat

Operand Calls Bytes Average Ratio %

----------------------------------------------------------
ERROR 0 0 0 0
SET 354 51404 145 33
RENAME 0 0 0 0
REFRESH 1359 70668 52 46
DELETE 290 10440 36 6
SLINK 193 12352 64 8
UNLINK 0 0 0 0
MODIFYFIELDS 91 7280 80 4
RECORD DATA CONN 0 0 0 0
COMPLETE DATA CONN 0 0 0 0

Total bytes sent: 161292 (0 MB) in 1797 packets. Average 89

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      233

 Viewing Cluster IP Addresses

Viewing Cluster IP Addresses

Description
This command lets you see the IP addresses and interfaces of the Cluster Members.

Syntax

Shell Command

Gaia Clish show cluster members ips

Expert mode cphaprob tablestat

Example

Note - To see name of interfaces that correspond to numbers in the "Interface"

column, run the fw ctl iflist command.

[Expert@Member1:0]# cphaprob tablestat

---- Unique IP's Table ----

Member Interface IP-Address

------------------------------------------

(Local)
0 1 192.168.3.245
0 2 11.22.33.245
0 3 44.55.66.245

1 1 192.168.3.246
1 2 11.22.33.246
1 3 44.55.66.246

------------------------------------------

[Expert@Member1:0]#
[Expert@Member1:0]# fw ctl iflist
1 : eth0
2 : eth1
3 : eth2
[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      234

 Viewing the Cluster Member ID Mode in Local Logs

Viewing the Cluster Member ID Mode in Local Logs

Description
This command lets you see how the local ClusterXL logs show the Cluster Member - by its Member ID
(default), or its Member Name.
See "Configuring the Cluster Member ID Mode in Local Logs" on page 180.

Syntax

Shell Command

Gaia Clish show cluster members idmode

Expert mode cphaprob names

Example

[Expert@Member1:0]# cphaprob names 

Current member print mode in local logs is set to: ID

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      235

 Viewing Interfaces Monitored by RouteD

Viewing Interfaces Monitored by RouteD

Description
This command lets you see the interfaces, which the RouteD daemon monitors on the Cluster Member
when you configure OSPF.
The idea is that if you configure OSPF, Cluster Member monitors these interfaces and does not bring up
the Cluster Member unless RouteD daemon says it is OK to bring up the Cluster Member. This is used
mainly in ClusterXL High Availability Primary Up configuration to avoid premature failbacks.

Syntax

Shell Command

Gaia Clish show ospf interfaces [detailed]

Expert mode cphaprob routedifcs

Example 1

[Expert@Member1:0]# cphaprob routedifcs 

No interfaces are registered.

[Expert@Member1:0]#

Example 2

[Expert@Member1:0]# cphaprob routedifcs 

Monitored interfaces registered by routed:

eth0
[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      236

 Viewing Roles of RouteD Daemon on Cluster Members

Viewing Roles of RouteD Daemon on Cluster Members

Description
This command lets you view on which Cluster Member the RouteD daemon runs as a Master.

Notes:
n In ClusterXL High Availability, the RouteD daemon must run as a Master only on
the Active Cluster Member.
n In ClusterXL Load Sharing, the RouteD daemon must run as a Master only on
one of the Active Cluster Members and as a Non-Master on all other Cluster
Members.
n In VRRP Cluster, the RouteD daemon must run as a Master only on the VRRP
Master Cluster Member.

Syntax

Shell Command

Gaia Clish show cluster role

Expert mode cphaprob roles

Example

[Expert@Member1:0]# cphaprob roles

ID Role

1 (local) Master
2 Non-Master

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      237

 Viewing Cluster Correction Statistics

Viewing Cluster Correction Statistics

Description
This command lets you view the Cluster Correction Statistics on each Cluster Member.
The Cluster Correction Layer (CCL) is a mechanism that deals with asymmetric connections.
The CCL provides connections stickiness by "correcting" the packets to the correct Cluster Member:
n In most cases, the CCL makes the correction from the CoreXL SND.
n In some cases (like Dynamic Routing, or VPN), the CCL makes the correction from the Firewall or
SecureXL.
In some cases, ClusterXL needs to send some data along with the corrected packet (currently, only in
VPN). For such packets, the output shows "with metadata".

Note - For more information about CoreXL, see the R81 Performance Tuning
Administration Guide.

Syntax

Shell Command

Gaia Clish N/A

Expert mode cphaprob [{-d | -f | -s}] corr

Where:

Command Description

cphaprob corr Shows Cluster Correction Statistics for all traffic.

cphaprob -d corr Shows Cluster Correction Statistics for CoreXL SND only.

cphaprob -f corr Shows Cluster Correction Statistics for CoreXL Firewall instances only.

cphaprob -s corr Shows Cluster Correction Statistics for SecureXL only.

ClusterXL R81 Administration Guide      |      238

 Viewing Cluster Correction Statistics

Example 1 - For all traffic

[Expert@Member1:0]# cphaprob corr

Getting stats for SXL device 0, may take a few seconds...

Cluster Correction Stats (All Traffic):

------------------------------------------------------
Sent packets: 156 (0 with metadata)
Sent bytes: 34,568
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
Local asymmetric conns: 0
[Expert@Member1:0]#

Example 2 - For CoreXL SND only

[Expert@Member1:0]# cphaprob -d corr

Cluster Correction Stats (Dispatcher Corrections only):

------------------------------------------------------
Sent packets: 0 (0 with metadata)
Sent bytes: 0
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
[Expert@Member1:0]#

Example 3 - For CoreXL Firewall instances only

[Expert@Member1:0]# cphaprob -f corr

Cluster Correction Stats (Firewall instances only):

------------------------------------------------------
Sent packets: 156 (0 with metadata)
Sent bytes: 34,568
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
Local asymmetric conns: 0
[Expert@Member1:0]#

Example 4 - For SecureXL only

[Expert@Member1:0]# cphaprob -s corr

Getting stats for SXL device 0, may take a few seconds...

Cluster Correction Stats (SXL Devices only):

------------------------------------------------------
Sent packets: 0 (0 with metadata)
Sent bytes: 0
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
Local asymmetric conns: 0
[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      239

 Viewing the Cluster Control Protocol (CCP) Settings

Viewing the Cluster Control Protocol (CCP) Settings

Description
n You can view the Cluster Control Protocol (CCP) mode on the Cluster Members.
n You can view the Cluster Control Protocol (CCP) Encryption on the Cluster Members - enabled or
disabled (and the encryption key).
See "Configuring the Cluster Control Protocol (CCP) Settings" on page 188

Syntax for viewing the Cluster Control Protocol (CCP) mode

Shell Command

Gaia Clish show cluster members interfaces virtual

Expert mode cphaprob -a if

Important - In R81, the CCP always runs in the unicast mode.

Syntax for viewing the Cluster Control Protocol (CCP) Encryption

Shell Command

Gaia Clish show cluster members ccpenc

Expert mode cphaprob ccp_encrypt

cphaprob ccp_encrypt_key

ClusterXL R81 Administration Guide      |      240

 Viewing Latency and Drop Rate of Interfaces

Viewing Latency and Drop Rate of Interfaces

Description
This command lets you see the latency and the drop rate of each interface.

Syntax

Shell Command

Gaia Clish N/A

Expert mode cphaprob latency

Example

[Expert@Member1:0]# cphaprob latency

id 2
Latency | Drop
[msec] | rate

eth0 0.000 0%
eth1 0.000 0%
eth2 0.000 0%

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      241

 Viewing the State of the Multi-Version Cluster Mechanism

Viewing the State of the Multi-Version Cluster Mechanism

Description
This command lets you see the state of the Multi-Version Cluster (MVC) Mechanism - enabled (ON) or
disabled (OFF).

See "Configuring the Multi-Version Cluster Mechanism" on page 196.

Syntax

Shell Command

Gaia Clish show cluster members mvc

Expert mode cphaprob mvc

Example

Member1> show cluster members mvc

ON

Member1>

ClusterXL R81 Administration Guide      |      242

 Viewing Full Connectivity Upgrade Statistics

Viewing Full Connectivity Upgrade Statistics

Description
This command lets you see the Full Connectivity Upgrade statistics when you upgrade between minor
versions.

Syntax

Shell Command

Gaia Clish N/A

Expert mode cphaprob fcustat

Example

[Expert@Member1:0]# cphaprob fcustat

During FCU....................... no
Connection module map............ none

Table id map (remote->local)..... none

Table handlers ..................

8151 --> 0x0x7f97c421d860 (sip_state)
8158 --> 0x0x7f97c43d8e30 (connections)
LD handlers......................
ok - 0
failed - 0

Global handlers ................. none

[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      243

 Monitoring Cluster Status in SmartConsole

Monitoring Cluster Status in SmartConsole

Background
To see the applicable logs in SmartConsole:
n From the left navigation panel, click Logs & Monitor > Logs .
To get logs about changes in the state of Cluster Members:
1. From the left navigation panel, click Gateways & Servers .
2. Open the cluster object.
3. From the left tree, click ClusterXL and VRRP.
4. Open the cluster object.
5. In the Tracking field, select Log.
6. Click OK.
7. Install the Access Control Policy on the cluster object.

ClusterXL Log Messages

This section uses these conventions:
1. Square brackets are used to indicate place holders, which are substituted by relevant data when an
actual log message is issued (for example, "[NUMBER]" is replaced by a numeric value).
2. Angle brackets are used to indicate alternatives, one of which is used in actual log messages.
The different alternatives are separated with a vertical line (for example, "<up|down>" indicates
that either "up" or "down" is used).
3. These placeholders are frequently used:
n ID: A unique Cluster Member identifier, starting from "1". This corresponds to the order, in
which Cluster Members are sorted in the cluster object on the Cluster Members page.
n IP: Any unique IP address that belongs to the Cluster Member.
n MODE: The cluster mode (for example, New HA, LS Multicast, and so on).
n STATE: The state of the member (for example, active, down, standby ).
n DEVICE: The name of a Critical Device (for example, Interface Active Check , fwd).

ClusterXL R81 Administration Guide      |      244

 Monitoring Cluster Status in SmartConsole

General Logs
Log Description

Starting <ClusterXL|State Indicates that ClusterXL (or State

Synchronization>. Synchronization, for 3rd party clusters) was
successfully started on the reporting
Cluster Member.
This message is usually issued after a
member boots, or after an explicit call to
cphastart.

Stopping <ClusterXL|State Informs that ClusterXL (or State

Synchronization>. Synchronization) was deactivated on this
Cluster Member.
The Cluster Member is no longer a part of
the cluster (even if configured to be so),
until ClusterXL is restarted.

Unconfigured cluster Computers changed This message is usually issued when a

their MAC Addresses. Please reboot the Cluster Member is shut down, or after an
cluster so that the changes take explicit call to the cphastop.
affect.

State Logs
Log Description

Mode inconsistency detected: This message should rarely happen.

member [ID] ([IP]) will change It indicates that another Cluster Member has
its mode to [MODE]. Please re- reported a different cluster mode than is known to the
install the security policy on local Cluster Member.
the cluster. This is usually the result of a failure to install the
Access Control Policy on all Cluster Members.
To correct this problem, install the Access Control
Policy again.
Note - The cluster continues to operate
after a mode inconsistency is detected by
altering the mode of the reporting Cluster
Member to match the other Cluster
Members. However, we highly recommend
to re-install the policy as soon as possible.

ClusterXL R81 Administration Guide      |      245

 Monitoring Cluster Status in SmartConsole

Log Description

State change of member [ID] When a Cluster Member needs to change its state
([IP]) from [STATE] to [STATE] (for example, when an Active Cluster Member
was cancelled, since all other encounters a problem and needs to change its state
members are down. Member remains to "Down"), it first queries the other Cluster Members
[STATE]. for their state.
If all other Cluster Members are down, this Cluster
Member cannot change its state to a non-active one
(otherwise the cluster cannot function).
Thus, the reporting Cluster Member continues to
function, despite its problem (and usually reports its
state as "Active(!)").

member [ID] ([IP]) <is active|is This message is issued whenever a Cluster Member
down|is stand-by|is changes its state.
initializing> ([REASON]). The log text specifies the new state of the Cluster
Member.

Critical Device Logs

Log Description

[DEVICE] on member [ID] The Critical Device is working normally.

([IP]) status OK
([REASON])

[DEVICE] on member [ID] Either an error was detected by the Critical Device, or the
([IP]) detected a problem Critical Device has not reported its state for a number of
([REASON]). seconds (as set by the "timeout" option of the Critical
Device)

[DEVICE] on member [ID] Indicates that the Critical Device has registered itself with the
([IP]) is initializing Critical Device mechanism, but has not yet determined its
([REASON]). state.

[DEVICE] on member [ID] This message should not normally appear.

([IP]) is in an unknown Contact Check Point Support.
state ([STATE ID])
([REASON]).

Interface Logs
Log Description

interface [INTERFACE NAME] of Indicates that this interface is working normally - it is

member [ID] ([IP]) is up. able to receive and transmit Cluster Control Protocol
(CCP) packets on the expected subnet.

ClusterXL R81 Administration Guide      |      246

 Monitoring Cluster Status in SmartConsole

Log Description

interface [INTERFACE NAME] of This message is issued whenever an interface

member [ID] ([IP]) is down encounters a problem, either in receiving or
(receive <up|down>, transmit transmitting Cluster Control Protocol (CCP) packets.
<up|down>). Note that in this case the interface may still be
working properly, as far as the OS is concerned, but
is unable to communicate with other Cluster
Members.

interface [INTERFACE NAME] of Indicates that a new interface was registered with
member [ID] ([IP]) was added. the Cluster Member (meaning that Cluster Control
Protocol (CCP) packets arriving on this interface).
Usually, this message is the result of activating an
interface (such as issuing the "ifconfig up"
command).
The interface is now included in the ClusterXL
reports (in the output of the applicable CLI
commands.
Note that the interface may still be reported as
"Disconnected", in case it was configured as
such for ClusterXL.

interface [INTERFACE NAME] of Indicates that an interface was detached from the
member [ID] ([IP}) was removed. Cluster Member, and is therefore no longer
monitored by ClusterXL.

Reason Strings
Log Description

member [ID] ([IP]) reports This text can be included in a Critical Device log message
more interfaces up. describing the reasons for a problem report: another Cluster
Member has more interfaces reported to be working, than the
local Cluster Member does.
Usually, this means that the local Cluster Member has a faulty
interface, and that its peer Cluster Member can do a better
job as a Cluster Member. The local Cluster Member changes
it state to "Down", leaving the peer Cluster Member specified
in the message to handle traffic.

member [ID] ([IP]) has This message is issued when Cluster Members in the same
more interfaces - check cluster have a different number of interfaces.
your disconnected A Cluster Member with fewer interfaces than the maximal
interfaces configuration number in the cluster (the reporting Cluster Member) may not
in the <discntd.if be working properly, as it is missing an interface required to
file|registry> operate against a cluster IP address, or a synchronization
network.
If some of the interfaces on the other Cluster Member are
redundant, and should not be monitored by ClusterXL, they
should be explicitly designated as "Non-Monitored". See
"Defining Non-Monitored Interfaces" on page 144.

ClusterXL R81 Administration Guide      |      247

 Monitoring Cluster Status in SmartConsole

Log Description

[NUMBER] interfaces ClusterXL has detected a problem with one or more of the
required, only [NUMBER] monitored interfaces.
up. This does not necessarily mean that the Cluster Member
changes its state to "Down", as the other Cluster Members
may have less operational interfaces.
In such a condition, the Cluster Member with the largest
number of operational interfaces will remain up, while the
others go down.

ClusterXL R81 Administration Guide      |      248

 Working with SNMP Traps

Working with SNMP Traps

You can configure and see SNMP traps for ClusterXL High Availability.

To configure an SNMP trap:

1. Connect to the command line on the Management Server.
2. Log in to the Expert mode.
3. On a Multi-Domain Server, go to the context of the applicable Domain Management Server:

mdsenv <IP Address or Name of Domain Management Server>

4. Run:

threshold_config

For more information, see the R81 CLI Reference Guide > Chapter Security Management Server
Commands > Section threshold_config.
5. From the Threshold Engine Configuration Options menu, select (9) Configure Thresholds .
6. From the Threshold Categories menu, select (2) High Availability .
7. Select the applicable traps.
8. Select and configure these actions for the specified trap:
n Enable/Disable Threshold
n Set Severity
n Set Repetitions
n Configure Alert Destinations
9. From the Threshold Engine Configuration Options menu, select (7) Configure alert destinations .
10. Configure your alert destinations.
11. From the Threshold Engine Configuration Options menu, select (3) Save policy .
You can optionally save the policy to a file.
12. In SmartConsole, install the Access Control Policy on this cluster object.

Note - You can download the most recent Check Point MIB files from sk90470.

ClusterXL R81 Administration Guide      |      249

 How to Initiate Cluster Failover

How to Initiate Cluster Failover

For more information on initiating manual cluster failovers, see sk55081.

Method 1 (recommended)

Command
Change in the State Command
in the Expert Notes
of the Cluster Member in Gaia Clish
Mode

Change the state of the set cluster clusterXL_ Does not disable
Cluster Member to DOWN member admin down admin down Delta Sync.

Change the state of the set cluster clusterXL_ Does not initiate
Cluster Member to UP member admin up admin up Full Sync.

See:
n "Initiating Manual Cluster Failover" on page 189
n "The clusterXL_admin Script" on page 273

Method 2 (not recommended)

Change in the State

Command in the Expert mode Notes
of the Cluster Member

Change the state of the 1. cphaconf set_pnote -d <Name of Does not

Cluster Member to DOWN Critical Device> -t 0 -s ok disable Delta
register Sync.
2. cphaconf set_pnote -d <Name of
Critical Device> -s problem
report

Change the state of the 1. cphaconf set_pnote -d <Name of Does not

Cluster Member to UP Critical Device> -s ok report initiate Full
2. cphaconf set_pnote -d <Name of Sync.
Critical Device> unregister

See:
n "Registering a Critical Device" on page 181
n "Reporting the State of a Critical Device" on page 184
n "Unregistering a Critical Device" on page 183

Notes:
n In Load Sharing mode, the cluster distributes the traffic load between the
remaining Active members.
n In High Availability mode, the cluster fails over to a Standby Cluster Member with
the highest priority.

ClusterXL R81 Administration Guide      |      250

 Troubleshooting Issues with the Critical Device "routed"

Troubleshooting Issues with the Critical Device

"routed"
Background
The Critical Device called routed monitors the state of the Dynamic Routing on the Cluster Member (see
"Viewing Critical Devices" on page 207).
This Critical Device makes sure that traffic is not assigned to a Cluster Member before it is ready to handle
the traffic.
The Gaia OS RouteD daemon handles all routing (static and dynamic) operations.
There can be an issue with Dynamic Routing with one or more of these symptoms:
n Cluster IP address connectivity problems
n Unexpected cluster failovers
n The state of the Critical Device routed is problem .
Example:

Device Name: routed

Registration number: 2
Timeout: none
Current state: problem
Time since last report: 10 sec

These are some of the common causes of this issue:

n Cluster misconfiguration
n Traffic on TCP port 2010 between Cluster Members is blocked
n The RouteD daemon did not get all of its routes
n The RouteD daemon did not start correctly

Standard Behavior of the Critical Device "routed"

Typically, the Critical Device routed reports its current state as problem when:
n A Cluster Member fails over
n A Cluster Member reboots
n There is an inconsistency in the Dynamic Routing configuration on Cluster Members
the Critical Device routed reports its current state as OK when
n A ClusterXL member tells the RouteD daemon that it is a Master
n The RouteD daemon gets the entire routing state from the Master

ClusterXL R81 Administration Guide      |      251

 Troubleshooting Issues with the Critical Device "routed"

Basic Troubleshooting Steps

1. Examine the cluster interfaces to make sure they are configured correctly.
See "Viewing Cluster Interfaces" on page 214.
2. In ClusterXL High Availability mode, make sure that the RouteD daemon is running on the Active
Cluster Member.
3. Make sure traffic on TCP port 2010 between the Cluster Members is not blocked.
4. Generate RouteD cluster messages.
Run this command in the Expert mode on the cluster member:

dbset routed:instance:default:traceoptions:traceoptions:Cluster

Examine the /var/log/routed/log file.

5. In ClusterXL High Availability mode with the OSPF configuration, make sure the OSPF interface is
up on the Standby Cluster Member.
6. In the OSPF configuration, look for a "router-id" mismatch.
For advanced troubleshooting procedures and more information, see sk92787.
For troubleshooting OSPF and the RouteD daemon, see sk84520.

ClusterXL R81 Administration Guide      |      252

 ClusterXL Error Messages

ClusterXL Error Messages

Each important cluster event that affects the Cluster Members has a unique code that appears in the
/var/log/messages file and dmesg.
Each cluster event message starts with the prefix CLUS-XXXXXX-Y.
Example: CLUS-214802-1

Part of the
Description
message

CLUS- Constant string.

XXXXXX A six-digit error code.

These error codes specify the events:
n Events related to Critical Devices
n Events related to the states of Cluster Members
n Events related to cluster synchronization
n Events related to policy installation

The first digit shows which Cluster Member generated the event:

n 1 - local member
n 2 - remote member

The second digit represents the cluster type event

The meaning of the other digits depends on the cluster event type.

Y Shows the ID or the NAME of the local Cluster Member that generated this log
message.
See "Configuring the Cluster Member ID Mode in Local Logs" on page 180.

For more information, see sk125152.

ClusterXL R81 Administration Guide      |      253

 Command Line Reference

Command Line Reference

See the R81 CLI Reference Guide.

ClusterXL R81 Administration Guide      |      254

 ClusterXL Configuration Commands

ClusterXL Configuration Commands

Description
These commands let you configure internal behavior of the Clustering Mechanism.

Important:
n We do not recommend that you run these commands. These commands must
be run automatically only by the Security Gateway or the Check Point Support.
n In a Cluster, you must configure all the Cluster Members in the same way.

Syntax

Notes:
n In Gaia Clish:
Enter the set cluster<ESC><ESC> to see all the available commands.
n In Expert mode:
Run the cphaconf command see all the available commands.
You can run the cphaconf commands only from the Expert mode.
n Syntax legend:
1. Curly brackets or braces { }:
Enclose a list of available commands or parameters, separated by the
vertical bar |, from which user can enter only one.
2. Angle brackets < > :
Enclose a variable - a supported value user needs to specify explicitly.
3. Square brackets or brackets [ ]:
Enclose an optional command or parameter, which user can also enter.
n You can include these commands in scripts to run them automatically.
The meaning of each command is explained in the next sections.
Table: ClusterXL Configuration Commands
Description Command in Command in
of Command Gaia Clish Expert Mode

Configure how to show the Cluster Member in set cphaconf mem_id_mode {id
local ClusterXL logs - by its Member ID or its cluster | name}
Member Name (see "Configuring the Cluster member
Member ID Mode in Local Logs" on page 180) idmode {id
| name}

Register a single Critical Device (Pnote) on the N / A cphaconf set_pnote -d

Cluster Member (see "Registering a Critical <Name of Device> -t
Device" on page 181) <Timeout in Sec> -s
{ok|init|problem} [-p]
[-g] register

Unregister a single Critical Device (Pnote) on N / A cphaconf set_pnote -d

the Cluster Member (see "Unregistering a <Name of Device> [-p] [-
Critical Device" on page 183) g] unregister

ClusterXL R81 Administration Guide      |      255

 ClusterXL Configuration Commands

Table: ClusterXL Configuration Commands (continued)

Description Command in Command in
of Command Gaia Clish Expert Mode

Report (change) a state in a single Critical N / A cphaconf set_pnote -d

Device (Pnote) on the Cluster Member (see <Name of Device> -s
"Reporting the State of a Critical Device" on {ok|init|problem} [-g]
page 184) report

Register several Critical Devices (Pnotes) from N / A cphaconf set_pnote -f

a file on the Cluster Member (see "Registering <Name of File> [-g]
Critical Devices Listed in a File" on page 185) register

Unregister all Critical Devices (Pnotes) on the N / A cphaconf set_pnote -a [-

Cluster Member (see "Unregistering All Critical g] unregister
Devices" on page 187)

Configure the Cluster Control Protocol (CCP) set cphaconf ccp_encrypt

Encryption on the Cluster Member (see cluster {off | on}
"Configuring the Cluster Control Protocol member cphaconf ccp_encrypt_key
(CCP) Settings" on page 188) ccpenc <Key String>
{off | on}

Configure the Cluster Forwarding Layer on the set cphaconf forward {off |
Cluster Member (controls the forwarding of cluster on}
traffic between Cluster Members) member
Note - For Check Point use only. forwarding
{off | on}

Print the current cluster configuration as loaded N / A cphaconf debug_data

in the kernel on the Cluster Member (for details,
see sk93306)

Start internal failover between slave interfaces N / A cphaconf failover_bond

of specified bond interface - only in Bond High <bond_name>
Availability mode (for details, see sk93306)

Configure what happens during a failover after N / A cphaconf enable_bond_

a Bond already failed over internally (for details, failover <bond_name>
see sk93306)

Initiate manual cluster failover (see "Initiating set clusterXL_admin {down |

Manual Cluster Failover" on page 189) cluster up}
member
admin
{down |
up}

Configure the minimal number of required N / A cphaconf bond_ls {set

slaves interfaces for Bond Load Sharing (see <Bond Name> <Value> |
"Configuring the Minimal Number of Required remove <Bond Name>}
Slave Interfaces for Bond Load Sharing" on
page 193)

ClusterXL R81 Administration Guide      |      256

 ClusterXL Configuration Commands

Table: ClusterXL Configuration Commands (continued)

Description Command in Command in
of Command Gaia Clish Expert Mode

Configuring Link Monitoring on the Cluster N / A N / A

Interfaces (see "Configuring Link Monitoring on
the Cluster Interfaces" on page 107)

Configuring the Multi-Version Cluster N / A cphaconf mvc {off | on}

Mechanism (see "Configuring the Multi-
Version Cluster Mechanism" on page 196)

List of the Gaia Clish set cluster member commands

set cluster member admin {down | up} [permanent]

set cluster member ccpenc {off | on}
set cluster member forwarding {off | on}
set cluster member idmode {id | name}
set cluster member mvc {off | on}

List of the cphaconf commands

Note - Some commands are not applicable to 3rd party clusters.

cphaconf [-D] <options> start

cphaconf stop
cphaconf [-t <Sync IF 1>...] [-d <Non-Monitored IF 1>...] add
cphaconf clear-secured
cphaconf clear-non-monitored
cphaconf debug_data
cphaconf delete_link_local [-vs <VSID>] <IF name>
cphaconf set_link_local [-vs <VSID>] <IF name> <Cluster IP>
cphaconf mem_id_mode {id | name}
cphaconf failover_bond <bond_name>
cphaconf [-s] {set | unset | get} var <Kernel Parameter Name>
[<Value>]
cphaconf bond_ls {set <Bond Name> <Value> | remove <Bond Name>}
cphaconf set_pnote -d <Device> -t <Timeout in sec> -s {ok | init |
problem} [-p] [-g] register
cphaconf set_pnote -f <File> [-g] register
cphaconf set_pnote -d <Device> [-p] [-g] unregister
cphaconf set_pnote -a [-g] unregister
cphaconf set_pnote -d <Device> -s {ok | init | problem} [-g] report
cphaconf ccp_encrypt {off | on}
cphaconf ccp_encrypt_key <Key String>

ClusterXL R81 Administration Guide      |      257

 ClusterXL Monitoring Commands

ClusterXL Monitoring Commands

Description
Use the monitoring commands to make sure that the cluster and the Cluster Members work properly, and
to define Critical Devices. A Critical Device (also known as a Problem Notification, or pnote) is a special
software device on each Cluster Member, through which the critical aspects for cluster operation are
monitored. When the critical monitored component on a Cluster Member fails to report its state on time, or
when its state is reported as problematic, the state of that member is immediately changed to 'Down'.

Syntax

Notes:
n In Gaia Clish:
Enter the show cluster<ESC><ESC> to see all the available commands.
n In Expert mode:
Run the cphaprob command see all the available commands.
You can run the cphaprob commands from Gaia Clish as well.
n Syntax legend:
1. Curly brackets or braces { }:
Enclose a list of available commands or parameters, separated by the
vertical bar |, from which user can enter only one.
2. Angle brackets < > :
Enclose a variable - a supported value user needs to specify explicitly.
3. Square brackets or brackets [ ]:
Enclose an optional command or parameter, which user can also enter.
n You can include these commands in scripts to run them automatically.
The meaning of each command is explained in the next sections.
Table: ClusterXL Monitoring Commands
Description Command in Command in
of Command Gaia Clish Expert Mode

Show states of Cluster Members and their names (see show cluster cphaprob [-
"Viewing Cluster State" on page 203) state vs <VSID>]
state

Show Critical Devices (Pnotes) and their states on the show cluster cphaprob [-
Cluster Member (see "Viewing Critical Devices" on members pnotes l] [-ia] [-
page 207) {all | problem} e] list

Show cluster interfaces on the cluster member (see show cluster cphaprob [-
"Viewing Cluster Interfaces" on page 214) members vs all] [-a]
interfaces {all [-m] if
| secured |
virtual |
vlans}

ClusterXL R81 Administration Guide      |      258

 ClusterXL Monitoring Commands

Table: ClusterXL Monitoring Commands (continued)

Description Command in Command in
of Command Gaia Clish Expert Mode

Show cluster bond configuration on the Cluster Member show cluster cphaprob
(see "Viewing Bond Interfaces" on page 218) bond {all | show_bond
name <bond_ [<bond_
name>} name>]

Show groups of bonds on the Cluster Member (see N / A cphaprob

"Viewing Bond Interfaces" on page 218) show_bond_
groups

Show (and reset) cluster failover statistics on the show cluster cphaprob [-
Cluster Member (see "Viewing Cluster Failover failover [reset reset {-c |
Statistics" on page 223) {count | -h}] [-l
history}] <count>]
show_
failover

Show information about the software version (including show cluster cphaprob
hotfixes) on the local Cluster Member and its release release
matches/mismatches with other Cluster Members (see
"Viewing Software Versions on Cluster Members" on
page 225)

Show Delta Sync statistics on the Cluster Member (see show cluster cphaprob [-
"Viewing Delta Synchronization" on page 226) statistics sync reset]
[reset] syncstat

Show Delta Sync statistics for the Connections table on show cluster cphaprob [-
the Cluster Member (see "Viewing Cluster Delta Sync statistics reset]
Statistics for Connections Table" on page 233) transport ldstat
[reset]

Show the Cluster Control Protocol (CCP) mode on the show cluster cphaprob [-
Cluster Member (see "Viewing Cluster Interfaces" on members vs all] -a
page 214) interfaces if
virtual

Show the IGMP membership of the Cluster Member show cluster cphaprob
(see "Viewing IGMP Status" on page 232) members igmp igmp

Show cluster unique IP's table on the Cluster Member show cluster cphaprob
(see "Viewing Cluster IP Addresses" on page 234) members ips tablestat

Show the Cluster Member ID Mode in local logs - by show cluster cphaprob
Member ID (default) or Member Name (see "Viewing members idmode names
the Cluster Member ID Mode in Local Logs" on
page 235)

ClusterXL R81 Administration Guide      |      259

 ClusterXL Monitoring Commands

Table: ClusterXL Monitoring Commands (continued)

Description Command in Command in
of Command Gaia Clish Expert Mode

Show interfaces, which the RouteD monitors on the show ospf cphaprob
Cluster Member when you configure OSPF (see interfaces routedifcs
"Viewing Interfaces Monitored by RouteD" on [detailed]
page 236)

Show roles of RouteD daemon on Cluster Members show cluster cphaprob

(see "Viewing Roles of RouteD Daemon on Cluster roles roles
Members" on page 237)

Show Cluster Correction Statistics (see "Viewing N / A cphaprob [{-

Cluster Correction Statistics" on page 238) d | -f | -
s}] corr

Show the Cluster Control Protocol (CCP) mode (see show cluster cphaprob -a
"Viewing the Cluster Control Protocol (CCP) Settings" members if
on page 240) interfaces
virtual

Show the Cluster Control Protocol (CCP) Encryption show cluster cphaprob
settings (see "Viewing the Cluster Control Protocol members ccpenc ccp_encrypt
(CCP) Settings" on page 240)

Shows the state of the Multi-Version Cluster (see show cluster N / A

"Viewing the State of the Multi-Version Cluster members mvc
Mechanism" on page 242)

Shows the latency and the drop rate of each interface N / A N / A

(see "Viewing Latency and Drop Rate of Interfaces" on
page 241)

Show Full Connectivity Upgrade statistics (see "Viewing N / A cphaprob

Full Connectivity Upgrade Statistics" on page 243) fcustat

ClusterXL R81 Administration Guide      |      260

 ClusterXL Monitoring Commands

List of the Gaia Clish show cluster commands

show cluster
      bond
            all
            name <Name of Bond>
      failover
      members
            ccpenc
            idmode
            igmp
            interfaces
                  all
                  secured
                  virtual
                  vlans
            ips
            mvc
            pnotes
                  all
                  problem
      release
      roles
      state
      statistics
            sync [reset]
            transport [reset]

ClusterXL R81 Administration Guide      |      261

 ClusterXL Monitoring Commands

List of the cphaprob commands

Note - Some commands are not applicable to 3rd party clusters.

cphaprob [-vs <VSID>] state

cphaprob [-reset {-c | -h}] [-l <count>] show_failover
cphaprob names
cphaprob [-reset] [-a] syncstat
cphaprob [-reset] ldstat
cphaprob [-l] [-i[a]] [-e] list
cphaprob [-vs all] [-a] [-m] if
cphaprob latency
cphaprob show_bond [<bond_name>]
cphaprob show_bond_groups
cphaprob igmp
cphaprob fcustat
cphaprob tablestat
cphaprob routedifcs
cphaprob roles
cphaprob release
cphaprob ccp_encrypt
cphaprob [{-d | -f | -s}] corr

ClusterXL R81 Administration Guide      |      262

 cpconfig

cpconfig
Description
This command starts the Check Point Configuration Tool.
This tool lets you configure specific settings for the installed Check Point products.

Important - In a Cluster, you must configure all the Cluster Members in the same way.

Syntax

cpconfig

Menu Options

Note - The options shown depend on the configuration and installed products.

Menu Option Description

Licenses and contracts Manages Check Point licenses and contracts on this Security
Gateway or Cluster Member.

SNMP Extension Obsolete. Do not use this option anymore.

To configure SNMP, see the R81 Gaia Administration Guide -
Chapter System Management - Section SNMP.

PKCS#11 Token Register a cryptographic token, for use by Gaia Operating

System.
See details of the token, and test its functionality.

Random Pool Configures the RSA keys, to be used by Gaia Operating System.

Secure Internal Communication Manages SIC on the Security Gateway or Cluster Member.
This change requires a restart of Check Point services on the
Security Gateway or Cluster Member.
For more information, see:
n The R81 Security Management Administration Guide.
n sk65764: How to reset SIC.

Enable cluster membership for Enables the cluster membership on the Security Gateway.
this gateway This change requires a reboot of the Security Gateway.
For more information, see the R81 Installation and Upgrade
Guide.

ClusterXL R81 Administration Guide      |      263

 cpconfig

Menu Option Description

Disable cluster membership for Disables the cluster membership on the Security Gateway.
this gateway This change requires a reboot of the Security Gateway.
For more information, see the R81 Installation and Upgrade
Guide.

Enable Check Point Per Virtual Enables Virtual System Load Sharing on the VSX Cluster
System State Member.
For more information, see the R81 VSX Administration Guide.

Disable Check Point Per Virtual Disables Virtual System Load Sharing on the VSX Cluster
System State Member.
For more information, see the R81 VSX Administration Guide.

Enable Check Point ClusterXL Enables Check Point ClusterXL for Bridge mode.
for Bridge Active/Standby This change requires a reboot of the Cluster Member.
For more information, see the R81 Installation and Upgrade
Guide.

Disable Check Point ClusterXL Disables Check Point ClusterXL for Bridge mode.
for Bridge Active/Standby This change requires a reboot of the Cluster Member.
For more information, see the R81 Installation and Upgrade
Guide.

Check Point CoreXL Manages CoreXL on the Security Gateway or Cluster Member.
After all changes in CoreXL configuration, you must reboot the
Security Gateway or Cluster Member.
For more information, see the R81 Performance Tuning
Administration Guide.

Automatic start of Check Point Shows and controls which of the installed Check Point products
Products start automatically during boot.

Exit Exits from the Check Point Configuration Tool.

ClusterXL R81 Administration Guide      |      264

 cpconfig

Example 1 - Menu on a single Security Gateway

[Expert@MySingleGW:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Enable cluster membership for this gateway
(7) Check Point CoreXL
(8) Automatic start of Check Point Products

(9) Exit

Enter your choice (1-9) :

Example 2 - Menu on a Cluster Member

[Expert@MyClusterMember:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Check Point CoreXL
(10) Automatic start of Check Point Products

(11) Exit

Enter your choice (1-11) :

ClusterXL R81 Administration Guide      |      265

 cphastart

cphastart
Description
Starts the cluster configuration on a Cluster Member after it was stopped with the "cphastop" on page 267
command.

Best Practice - To start a Cluster Member, use the "cpstart" command.

Note - This command does not initiate a Full Synchronization on the Cluster Member.

Syntax

cphastart
[-h]
[-d]

Parameters

Parameter Description

-h Shows the applicable built-in usage.

-d Runs the command in debug mode.

Use only if you troubleshoot the command itself.

Best Practice - If you use this parameter, then redirect the output to a file, or
use the script command to save the entire CLI session.

Refer to:
n These lines in the output file:
prepare_command_args: -D ... start
/opt/CPsuite-R81/fw1/bin/cphaconf clear-secured
/opt/CPsuite-R81/fw1/bin/cphaconf -D ...(truncated here
for brevity)... start
n The $FWDIR/log/cphastart.elg log file.

ClusterXL R81 Administration Guide      |      266

 cphastop

cphastop
Description
Stops the cluster software on a Cluster Member.

Best Practice - To stop a Cluster Member, use the "cpstop" command.

Notes:
n This command stops the Cluster Member from passing traffic.
n This command stops the State Synchronization between this Cluster Member
and its peer Cluster Members.
n After you run this command, you can still open connections directly to this
Cluster Member.
n To start the cluster software, run the "cphastart" on page 266 command.

Syntax

cphastop

ClusterXL R81 Administration Guide      |      267

 cp_conf fullha

cp_conf fullha
Description
Manages the state of the Full High Availability Cluster:
n Enables the Full High Availability Cluster
n Disables the Full High Availability Cluster
n Deletes the Full High Availability peer
n Shows the Full High Availability state

Important - To configure a Full High Availability cluster, follow the R81 Installation and
Upgrade Guide.

Syntax

cp_conf fullha
      enable
      del_peer
      disable
      state

Parameters

Parameter Description

enable Enables the Full High Availability on this computer.

del_peer Deletes the Full High Availability peer from the configuration.

disable Disables the Full High Availability on this computer.

state Shows the Full High Availability state on this computer.

Example

[Expert@Cluster_Member:0]# cp_conf fullha state

FullHA is currently enabled
[Expert@Cluster_Member:0]#

ClusterXL R81 Administration Guide      |      268

 cp_conf ha

cp_conf ha
Description
Enables or disables cluster membership on this Security Gateway.

Important - This command is for Check Point use only. To configure cluster
membership, you must use the "cpconfig" command.

Syntax

cp_conf ha {enable | disable} [norestart]

Parameters

Parameter Description

enable Enables cluster membership on this Security Gateway.

This command is equivalent to the option Enable cluster membership for this
gateway in the "cpconfig" menu.

disable Disables cluster membership on this Security Gateway.

This command is equivalent to the option Disable cluster membership for this
gateway in the "cpconfig" menu.

norestart Optional: Specifies to apply the configuration change without the restart of Check Point
services. The new configuration takes effect only after reboot.

Example 1 - Enable the cluster membership without restart of Check Point services

[Expert@MyGW:0]# cp_conf ha enable norestart

Cluster membership for this gateway was enabled successfully

Important: This change will take effect after reboot.

[Expert@MyGW:0]#

Example 2 - Disable the cluster membership without restart of Check Point services

[Expert@MyGW:0]# cp_conf ha disable norestart

cpwd_admin:
Process CPHAMCSET process has been already terminated

Cluster membership for this gateway was disabled successfully

Important: This change will take effect after reboot.

[Expert@MyGW:0]#

ClusterXL R81 Administration Guide      |      269

 fw hastat

fw hastat
Description
Shows information about Check Point computers in High Availability configuration and their states.

Note - This command is outdated. On cluster members, run the Gaia Clish command
"show cluster state", or the Expert mode command "cphaprob state". See
"Viewing Cluster State" on page 203.

Syntax

fw hastat [<Target1>] [<Target2>] ... [<TargetN>]

Parameters

Parameter Description

<Target1> Specifies the Check Point computers to query.

<Target2> ... If you run this command on the Management Server, you can enter the applicable IP
<TargetN> address, or the resolvable HostName of the managed Security Gateway or Cluster
Member.
If you do not specify the target, the command queries the local computer.

Example - Querying the local Cluster Member

[Expert@Member1:0]# fw hastat
HOST NUMBER HIGH AVAILABILITY STATE MACHINE STATUS
192.168.3.52 1 active OK
[Expert@Member1:0]#

ClusterXL R81 Administration Guide      |      270

 fwboot ha_conf

fwboot ha_conf
Description
Configures the cluster mechanism during boot.

Important - This command is for Check Point use only.

Notes:
n You must run this command from the Expert mode.
n To install a cluster, see the R81 Installation and Upgrade
Guide.

Syntax

[Expert@HostName:0]# $FWDIR/boot/fwboot ha_conf

ClusterXL R81 Administration Guide      |      271

 ClusterXL Scripts

ClusterXL Scripts
You can use special scripts to change the state of Cluster Members.

ClusterXL R81 Administration Guide      |      272

 The clusterXL_admin Script

The clusterXL_admin Script

Description
You can use the clusterXL_admin script to initiate a manual fail-over from a Cluster Member.
Location of this script on your Cluster Members is:

$FWDIR/bin/clusterXL_admin

Script Workflow
This shell script does one of these:
n Registers a Critical Device called "admin_down" and reports the state of that Critical Device as
"problem".
This gracefully changes the state of the Cluster Member to "DOWN".
n Reports the state of the registered Critical Device "admin_down" as "ok".
This gracefully changes the state of the Cluster Member to "UP".
Then, the script unregisters the Critical Device "admin_down".

For more information, see sk55081.

ClusterXL R81 Administration Guide      |      273

 The clusterXL_admin Script

Example

ClusterXL R81 Administration Guide      |      274

 The clusterXL_admin Script

#! /bin/csh -f
#
# The script will cause the machine to get into down state, thus the member will not filter packets.
# It will supply a simple way to initiate a failover by registering a new device in problem state when
# a failover is required and will unregister the device when wanting to return to normal operation.
# USAGE:
# clusterXL_admin <up|down>

set PERSISTENT = ""

# checking number of arguments

if ( $#argv > 2 || $#argv < 1 ) then
echo "clusterXL_admin : Invalid Argument Count"
echo "Usage: clusterXL_admin <up|down> [-p]"
exit 1
else if ( "$1" != "up" && "$1" != "down" ) then
echo "clusterXL_admin : Invalid Argument ($1)"
echo "Usage: clusterXL_admin <up|down> [-p]"
exit 1
else if ( $#argv == 2 ) then
if ( "$2" != "-p" ) then
echo "clusterXL_admin : Invalid Argument ($2)"
echo "Usage: clusterXL_admin <up|down> [-p]"
exit 1
endif
set PERSISTENT = "-p"
endif

#checking if cpha is started

$FWDIR/bin/cphaprob stat | grep "Cluster" > /dev/null
if ($status) then
echo "HA is not started"
exit 1
endif

# Inform the user that the command can run with persistent mode.
if ("$PERSISTENT" != "-p") then
echo "This command does not survive reboot. To make the change permanent, please run 'set cluster
member admin down/up permanent' in clish or add '-p' at the end of the command in expert mode"
endif

if ( $1 == "up" ) then

echo "Setting member to normal operation ..."
$FWDIR/bin/cphaconf set_pnote -d admin_down $PERSISTENT unregister > & /dev/null
if ( `uname` == 'IPSO' ) then
sleep 5
else
sleep 1
endif

set stateArr = `$FWDIR/bin/cphaprob stat | grep "local"`

$FWDIR/bin/cphaprob stat | egrep "Sync only|Bridge Mode" > /dev/null

#If it's third party or bridge mode, use column 4 , otherwise 5
if ($status) then
set state = $stateArr[5]
else
set state = $stateArr[4]
endif

echo "Member current state is $state"

if (($state != "Active" && $state != "Standby") && ($state != "ACTIVE" && $state != "STANDBY" &&
$state != "ACTIVE(!)")) then
echo "Operation failed: member is still down, please run 'show cluster members pnotes
problem' in clish or 'cphaprob list' in expert mode for further details"
endif
exit 0
endif

if ( $1 == "down" ) then

echo "Setting member to administratively down state ..."
$FWDIR/bin/cphaconf set_pnote -d admin_down -t 0 -s problem $PERSISTENT register > & /dev/null

ClusterXL R81 Administration Guide      |      275

 The clusterXL_admin Script

sleep 1

set stateArr = `$FWDIR/bin/cphaprob stat | grep "local"`

$FWDIR/bin/cphaprob stat | egrep "Sync only|Bridge Mode" > /dev/null

#If it's third party or bridge mode, use column 4 , otherwise 5
if ($status) then
set state = $stateArr[5]
else
set state = $stateArr[4]
endif

echo "Member current state is $state"

if ( $state == "Active attention" || $state == "ACTIVE(!)" ) then
echo "All the members within the cluster have problem/s and the local member was chosen to
become active"
else
if ( $state != "Down" && $state != "DOWN" ) then
echo "Operation failed: member is still down, please run 'show cluster members
pnotes problem' in clish or 'cphaprob list' in expert mode for further details"
endif
endif
exit 0
else
echo "clusterXL_admin : Invalid Option ($1)"
echo "Usage: clusterXL_admin <up|down> [-p]"
exit 1
endif

ClusterXL R81 Administration Guide      |      276

 The clusterXL_monitor_ips Script

The clusterXL_monitor_ips Script

Description
You can use the clusterXL_monitor_ips script to ping a list of predefined IP addresses and change the
state of the Cluster Member to DOWN or UP based on the replies to these pings. For this script to work, you
must write the IP addresses in the $FWDIR/conf/cpha_hosts file - each IP address on a separate
line. This file does not support comments or spaces.
Location of this script on your Cluster Members is:

$FWDIR/bin/clusterXL_monitor_ips

Script Workflow
1. Registers a Critical Device called "host_monitor" with the status "ok".
2. Starts to send pings to the list of predefined IP addresses in the $FWDIR/conf/cpha_hosts file.
3. While the script receives responses to its pings, it does not change the status of that Critical Device.
4. If the script does not receive a response to even one ping, it reports the state of that Critical Device
as "problem".
This gracefully changes the state of the Cluster Member to DOWN.
If the script receives responses to its pings again, it changes the status of that Critical Device to "ok"
again.
For more information, see sk35780.

Important - You must do these changes on all Cluster Members.

ClusterXL R81 Administration Guide      |      277

 The clusterXL_monitor_ips Script

Example

ClusterXL R81 Administration Guide      |      278

 The clusterXL_monitor_ips Script

#!/bin/sh
#
# The script tries to ping the hosts written in the file $FWDIR/conf/cpha_hosts. The names (must be
resolveable) ot the IPs of the hosrs must be written in seperate lines.
# the file must not contain anything else.
# We ping the given hosts every number of seconds given as parameter to the script.
# USAGE:
# cpha_monitor_ips X silent
# where X is the number of seconds between loops over the IPs.
# if silent is set to 1, no messages will appear on the console
#
# We initially register a pnote named "host_monitor" in the problem notification mechanism
# when we detect that a host is not responding we report the pnote to be in "problem" state.
# when ping succeeds again - we report the pnote is OK.

silent=0

if [ -n "$2" ]; then

if [ $2 -le 1 ]; then
silent=$2
fi
fi
hostfile=$FWDIR/conf/cpha_hosts
arch=`uname -s`
if [ $arch = "Linux" ]
then
#system is linux
ping="ping -c 1 -w 1"
else
ping="ping"
fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -t 0 -s ok register
TRUE=1
while [ "$TRUE" ]
do
result=1
for hosts in `cat $hostfile`
do
if [ $silent = 0 ]
then
echo "pinging $hosts using command $ping $hosts"
fi
if [ $arch = "Linux" ]
then
$ping $hosts > /dev/null 2>&1
else
$ping $hosts $1 > /dev/null 2>&1
fi
status=$?
if [ $status = 0 ]
then
if [ $silent = 0 ]
then
echo " $hosts is alive"
fi
else
if [ $silent = 0 ]
then
echo " $hosts is not responding "
fi
result=0
fi
done
if [ $silent = 0 ]
then
echo "done pinging"
fi
if [ $result = 0 ]
then
if [ $silent = 0 ]
then
echo " Cluster member should be down!"

ClusterXL R81 Administration Guide      |      279

 The clusterXL_monitor_ips Script

fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -s problem report
else
if [ $silent = 0 ]
then
echo " Cluster member seems fine!"
fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -s ok report
fi
if [ "$silent" = 0 ]
then
echo "sleeping"
fi
sleep $1
echo "sleep $1"
done

ClusterXL R81 Administration Guide      |      280

 The clusterXL_monitor_process Script

The clusterXL_monitor_process Script

Description
You can use the clusterXL_monitor_process script to monitor if the specified user space processes run,
and cause cluster fail-over if these processes do not run. For this script to work, you must write the correct
case-sensitive names of the monitored processes in the $FWDIR/conf/cpha_proc_list file - each
process name on a separate line. This file does not support comments or spaces.
Location of this script on your Cluster Members is:

$FWDIR/bin/clusterXL_monitor_process

Script Workflow
1. Registers Critical Devices (with the status "ok") called as the names of the processes you specified
in the $FWDIR/conf/cpha_proc_list file.
2. While the script detects that the specified process runs, it does not change the status of the
corresponding Critical Device.
3. If the script detects that the specified process do not run anymore, it reports the state of the
corresponding Critical Device as "problem".
This gracefully changes the state of the Cluster Member to "DOWN".
If the script detects that the specified process runs again, it changes the status of the corresponding
Critical Device to "ok" again.

For more information, see sk92904.

Important - You must do these changes on all Cluster Members.

ClusterXL R81 Administration Guide      |      281

 The clusterXL_monitor_process Script

Example

ClusterXL R81 Administration Guide      |      282

 The clusterXL_monitor_process Script

#!/bin/sh
#
# This script monitors the existance of processes in the system. The process names should be written
# in the $FWDIR/conf/cpha_proc_list file one every line.
#
# USAGE :
# cpha_monitor_process X silent
# where X is the number of seconds between process probings.
# if silent is set to 1, no messages will appear on the console.
#
#
# We initially register a pnote for each of the monitored processes
# (process name must be up to 15 charachters) in the problem notification mechanism.
# when we detect that a process is missing we report the pnote to be in "problem" state.
# when the process is up again - we report the pnote is OK.

if [ "$2" -le 1 ]
then
silent=$2
else
silent=0
fi
if [ -f $FWDIR/conf/cpha_proc_list ]
then
procfile=$FWDIR/conf/cpha_proc_list
else
echo "No process file in $FWDIR/conf/cpha_proc_list "
exit 0
fi

arch=`uname -s`

for process in `cat $procfile`

do
$FWDIR/bin/cphaconf set_pnote -d $process -t 0 -s ok -p register > /dev/null 2>&1
done

while [ 1 ]
do

result=1

for process in `cat $procfile`

do
ps -ef | grep $process | grep -v grep > /dev/null 2>&1

status=$?

if [ $status = 0 ]
then
if [ $silent = 0 ]
then
echo " $process is alive"
fi
# echo "3, $FWDIR/bin/cphaconf set_pnote -d $process -s ok report"
$FWDIR/bin/cphaconf set_pnote -d $process -s ok report
else
if [ $silent = 0 ]
then
echo " $process is down"
fi

$FWDIR/bin/cphaconf set_pnote -d $process -s problem report

result=0
fi

done

if [ $result = 0 ]

then
if [ $silent = 0 ]

ClusterXL R81 Administration Guide      |      283

 The clusterXL_monitor_process Script

then
echo " One of the monitored processes is down!"
fi
else
if [ $silent = 0 ]
then
echo " All monitored processes are up "
fi

fi
if [ "$silent" = 0 ]
then
echo "sleeping"
fi

sleep $1

done

ClusterXL R81 Administration Guide      |      284

 Cluster Management APIs

Cluster Management APIs

Introduction
The purpose of Cluster APIs is to provide automation / orchestration of Check Point cluster in a way similar
to simple-gateway APIs.
These Cluster APIs support common cluster operations - such as creating a new cluster object, modifying
an existing cluster object (for example, adding or removing cluster members, manipulation of interfaces).
These Cluster APIs are called "simple" because they do not support all cluster object features.
For operations on cluster objects that are not provided by these APIs, use SmartConsole.

List of APIs
API Category API Description

Asynchronous add simple- Creates a new simple cluster object from scratch
cluster

set simple- Modifies an existing simple cluster object

cluster

Synchronous show simple- Shows an existing simple cluster object specified by its
cluster Name or UID

show simple- Shows all existing simple cluster objects

clusters

delete simple- Deletes an existing simple cluster object

cluster

API Examples
Example 1 - Adding a simple cluster object

API command:
Use this API to add a simple cluster object.

add simple-cluster

Once the API command finishes, and the session is published, a new cluster object appears in
SmartConsole.

ClusterXL R81 Administration Guide      |      285

 Cluster Management APIs

Prerequisite:
1. All Cluster Members must already be installed.
2. The applicable interfaces on each Cluster Member must already be configured.

Example description:
n A simple ClusterXL in High Availability mode called cluster1
n With two cluster members called member1 and member2
n With three interfaces: eth0 (external), eth1 (sync), and eth2 (internal)
n Only the Firewall Software Blade is enabled (the IPsec VPN blade is disabled)
n Cluster software version is R80.20

Example cluster object topology:

Interface Cluster Member1 Member2

eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 /

255.255.255.0 255.255.255.0 255.255.255.0

eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 /

255.255.255.0 255.255.255.0

eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 /

255.255.255.0 255.255.255.0 255.255.255.0

API example:

Important - In the API command you must use the same One-Time Password you
used on Cluster Members during the First Time Configuration Wizard.

ClusterXL R81 Administration Guide      |      286

 Cluster Management APIs

{
"name" : "cluster1",
"color" : "yellow",
"version" : "R80.20",
"ip-address" : "172.23.5.254",
"os-name" : "Gaia",
"cluster-mode" : "cluster-xl-ha",
"firewall" : true,
"vpn" : false,
"interfaces" : [ 
{
"name" : "eth0",
"ip-address" : "172.23.5.254",
"network-mask" : "255.255.255.0",
"interface-type" : "cluster",
"topology" : "EXTERNAL",
"anti-spoofing" : "true"
},
{
"name" : "eth1",
"interface-type" : "sync",
"topology" : "INTERNAL",
"topology-settings": {
"ip-address-behind-this-interface": "network defined by the interface ip and net mask",
"interface-leads-to-dmz": false
}
},
{
"name" : "eth2",
"ip-address" : "192.168.1.254",
"network-mask" : "255.255.255.0",
"interface-type" : "cluster",
"topology" : "INTERNAL",
"anti-spoofing" : "true",
"topology-settings": {
"ip-address-behind-this-interface": "network defined by the interface ip and net mask",
"interface-leads-to-dmz": false
}
}
],
"members" : [ {
"name" : "member1",
"one-time-password" : "abcd",
"ip-address" : "172.23.5.1",
"interfaces" : [
{
"name" : "eth0",
"ip-address" : "172.23.5.1",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth1",
"ip-address" : "1.1.1.1",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth2",
"ip-address" : "192.168.1.1",
"network-mask" : "255.255.255.0"
}
]
},
{
"name" : "member2",
"one-time-password" : "abcd",

ClusterXL R81 Administration Guide      |      287

 Cluster Management APIs

"ip-address" : "172.23.5.2",
"interfaces" : [ 
{
"name" : "eth0",
"ip-address" : "172.23.5.2",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth1",
"ip-address" : "1.1.1.2",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth2",
"ip-address" : "192.168.1.2",
"network-mask" : "255.255.255.0"
}
]
}
]
}

Example 2 - Modifying an existing cluster object - adding a cluster member

API command:
Use this API to add (scale up) Cluster Members.

set simple-cluster

Example description:
Adding a Cluster Member called member3.

Example cluster object topology:

Interface Cluster Member1 Member2 Member3

eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 / 172.23.5.3 /

255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0

eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 / 1.1.1.3 /

255.255.255.0 255.255.255.0 255.255.255.0

eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 / 192.168.1.3 /

255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0

API example:

ClusterXL R81 Administration Guide      |      288

 Cluster Management APIs

{
"name" : "cluster1",
"members" : { "add" :
{
"name" : "member3",
"ipv4-address" : "172.23.5.3",
"one-time-password" : "aaaa",
"interfaces" : [ 
{
"name" : "eth0",
"ip-address" : "172.23.5.3",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth1",
"ip-address" : "1.1.1.3",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth2",
"ip-address" : "192.168.1.3",
"network-mask" : "255.255.255.0"
}
]
}
}
}

Example 3 - Modifying an existing cluster object - removing a cluster member

API command:
Use this API to remove (scale down) Cluster Members.

set simple-cluster

Example description:
Removing a Cluster Member called member3.

Example cluster object topology:

Interface Cluster Member1 Member2 Member3

eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 / 172.23.5.3 /

255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0

eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 / 1.1.1.3 /

255.255.255.0 255.255.255.0 255.255.255.0

eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 / 192.168.1.3 /

255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0

ClusterXL R81 Administration Guide      |      289

 Cluster Management APIs

API example:

{
"name" : "cluster1",
"members" : { "remove" : "member3" }
}

Example 4 - Modifying an existing cluster object - adding a cluster interface

API command:
Use this API to add a cluster interface.

set simple-cluster

Example description:
Adding a cluster interface called eth3.

Example cluster object topology:

Interface Cluster Member1 Member2

eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 /

255.255.255.0 255.255.255.0 255.255.255.0

eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 /

255.255.255.0 255.255.255.0

eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 /

255.255.255.0 255.255.255.0 255.255.255.0

eth3 (Internal) 10.10.10.254 / 10.10.10.1 / 10.10.10.2 /

255.255.255.0 255.255.255.0 255.255.255.0

API example:

ClusterXL R81 Administration Guide      |      290

 Cluster Management APIs

{
"name" : "cluster1",
"interfaces" : { "add" :
{
"name" : "eth3",
"ip-address" : "10.10.10.254",
"ipv4-mask-length" : "24",
"interface-type" : "cluster",
"topology" : "INTERNAL",
"anti-spoofing" : "true"
}
},
"members" : { "update" :
[{ 
"name" : "member1" ,
"interfaces" :
{"name" : "eth3",
"ipv4-address" : "10.10.10.1",
"ipv4-network-mask" : "255.255.255.0"}
},
{ 
"name" : "member2" ,
"interfaces" :
{"name" : "eth3",
"ipv4-address" : "10.10.10.2",
"ipv4-network-mask" : "255.255.255.0"}
}
]}
}

Example 5 - Modifying an existing cluster object - removing a cluster interface

API command:
Use this API to remove a cluster interface.

set simple-cluster

Example description:
Removing a cluster interface called eth3.

Example cluster object topology:

Interface Cluster Member1 Member2

eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 /

255.255.255.0 255.255.255.0 255.255.255.0

eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 /

255.255.255.0 255.255.255.0

eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 /

255.255.255.0 255.255.255.0 255.255.255.0

eth3 (Internal) 10.10.10.254 / 10.10.10.1 / 10.10.10.2 /

255.255.255.0 255.255.255.0 255.255.255.0

ClusterXL R81 Administration Guide      |      291

 Cluster Management APIs

API example:

{
"name" : "cluster1",
"interfaces" : { "remove" : "eth3" }
}

Example 6 - Modifying an existing cluster object - changing settings of a cluster interface

API command:
Use this API to change settings of a cluster interface.

set simple-cluster

Example description:
Changing the IP address of the cluster interfaces called eth2 from 192.168.x.254 / 255.255.255.0 to
172.30.1.x / 255.255.255.0

Example cluster object topology:

Interface Cluster Member1 Member2

eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 /

255.255.255.0 255.255.255.0 255.255.255.0

eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 /

255.255.255.0 255.255.255.0

eth2 (Internal) From: From: From:

192.168.1.254 / 192.168.1.1 / 192.168.1.2 /
255.255.255.0 255.255.255.0 255.255.255.0
To: To: To:
172.30.1.254 / 172.30.1.1 / 172.30.1.2 /
255.255.255.0 255.255.255.0 255.255.255.0

API example:

ClusterXL R81 Administration Guide      |      292

 Cluster Management APIs

{
"name" : "cluster1",
"interfaces" : { "update" :
{
"name" : "eth2",
"ip-address" : "172.30.1.254",
"ipv4-mask-length" : "24",
"interface-type" : "cluster",
"topology" : "INTERNAL",
"anti-spoofing" : "true"
}
},
"members" : { "update" : [
{
"name" : "member1" ,
"interfaces" :
{"name" : "eth2",
"ipv4-address" : "172.30.1.1",
"ipv4-mask-length" : "24"}
},
{
"name" : "member2" ,
"interfaces" :
{"name" : "eth2",
"ipv4-address" : "172.30.1.2",
"ipv4-mask-length" : "24"}
}
]}
}

Example 7 - Modifying an existing cluster object - reestablishing SIC

API command:
Use this API to reestablish SIC with Cluster Members.

set simple-cluster

Prerequisite:
SIC must already be reset on the Cluster Members.

ClusterXL R81 Administration Guide      |      293

 Cluster Management APIs

API example:

Important - In the API command you must use the same One-Time Password you
used on Cluster Members during the SIC reset.

{
"name" : "cluster1",
"members" : { "update" :
[
{
"name" : "member1",
"one-time-password" : "aaaa"
},
{
"name" : "member2",
"one-time-password" : "aaaa"
}
]
}
}

Example 8 - Modifying an existing cluster object - enabling / disabling blades

API command:
Use this API to enable and disable Software Blades on Cluster Members.

set simple-cluster
Notes:
n To enable a Software Blade, set its value to true in the
API command.
n To disable a Software Blade, set its value to false in the
API command.

ClusterXL R81 Administration Guide      |      294

 Cluster Management APIs

API example:
To enable all Software Blades supported by the Cluster API:

{
"name" : "cluster1",
"vpn" : true,
"application-control" : true,
"url-filtering" : true,
"ips" : true,
"content-awareness" : true,
"anti-bot" : true,
"anti-virus" : true,
"threat-emulation" : true
}

To disable all Software Blades supported by the Cluster API:

{
"name" : "cluster1",
"vpn" : false,
"application-control" : false,
"url-filtering" : false,
"ips" : false,
"content-awareness" : false,
"anti-bot" : false,
"anti-virus" : false,
"threat-emulation" : false
}

Example 9 - Viewing an existing cluster object

API command:
Use this API to view a specific existing cluster object.

show simple-cluster

Note - By default, the output shows up to 50 configured cluster interfaces.

API example - request:

{
"limit-interfaces" : "10",
"name" : "cluster1"
}

API example - response:

ClusterXL R81 Administration Guide      |      295

 Cluster Management APIs

{
"uid": "e0ce560b-8a0a-4468-baa9-5f8eb2658b96",
"name": "cluster1",
"type": "simple-cluster",
"domain": {
"uid": "41e821a0-3720-11e3-aa6e-0800200c9fde",
"name": "SMC User",
"domain-type": "domain"
},
"meta-info": {
"lock": "unlocked",
"validation-state": "ok",
"last-modify-time": {
"posix": 1567417185885,
"iso-8601": "2019-09-02T12:39+0300"
},
"last-modifier": "aa",
"creation-time": {
"posix": 1567417140278,
"iso-8601": "2019-09-02T12:39+0300"
},
"creator": "aa"
},
"tags": [],
"read-only": false,
"comments": "",
"color": "yellow",
"icon": "NetworkObjects/cluster",
"groups": [],
"ipv4-address": "172.23.5.254",
"dynamic-ip": false,
"version": "R80.20",
"os-name": "Gaia",
"hardware": "Open server",
"firewall": true,
"firewall-settings": {
"auto-maximum-limit-for-concurrent-connections": true,
"maximum-limit-for-concurrent-connections": 25000,
"auto-calculate-connections-hash-table-size-and-memory-pool": true,
"connections-hash-size": 131072,
"memory-pool-size": 6,
"maximum-memory-pool-size": 30
},
"vpn": false,
"application-control": false,
"url-filtering": false,
"content-awareness": false,
"ips": false,
"anti-bot": false,
"anti-virus": false,
"threat-emulation": false,
"save-logs-locally": false,
"send-alerts-to-server": [
"harry-main-take-96"

ClusterXL R81 Administration Guide      |      296

 Cluster Management APIs

],
"send-logs-to-server": [
"harry-main-take-96"
],
"send-logs-to-backup-server": [],
"logs-settings": {
"rotate-log-by-file-size": false,
"rotate-log-file-size-threshold": 1000,
"rotate-log-on-schedule": false,
"alert-when-free-disk-space-below-metrics": "mbytes",
"alert-when-free-disk-space-below": true,
"alert-when-free-disk-space-below-threshold": 3000,
"alert-when-free-disk-space-below-type": "popup alert",
"delete-when-free-disk-space-below-metrics": "mbytes",
"delete-when-free-disk-space-below": true,
"delete-when-free-disk-space-below-threshold": 5000,
"before-delete-keep-logs-from-the-last-days": false,
"before-delete-keep-logs-from-the-last-days-threshold": 0,
"before-delete-run-script": false,
"before-delete-run-script-command": "",
"stop-logging-when-free-disk-space-below-metrics": "mbytes",
"stop-logging-when-free-disk-space-below": true,
"stop-logging-when-free-disk-space-below-threshold": 100,
"reject-connections-when-free-disk-space-below-threshold": false,
"reserve-for-packet-capture-metrics": "mbytes",
"reserve-for-packet-capture-threshold": 500,
"delete-index-files-when-index-size-above-metrics": "mbytes",
"delete-index-files-when-index-size-above": false,
"delete-index-files-when-index-size-above-threshold": 100000,
"delete-index-files-older-than-days": false,
"delete-index-files-older-than-days-threshold": 14,
"forward-logs-to-log-server": false,
"perform-log-rotate-before-log-forwarding": false,
"update-account-log-every": 3600,
"detect-new-citrix-ica-application-names": false,
"turn-on-qos-logging": true
},
"interfaces": {
"total": 3,
"from": 1,
"to": 3,
"objects": [
{
"name": "eth0",
"ipv4-address": "172.23.5.254",
"ipv4-network-mask": "255.255.255.0",
"ipv4-mask-length": 24,
"ipv6-address": "",
"topology": "external",
"anti-spoofing": true,
"anti-spoofing-settings": {
"action": "prevent"
},
"security-zone": false,
"comments": "",
"color": "black",
"icon": "NetworkObjects/network",
"interface-type": "cluster"
},

ClusterXL R81 Administration Guide      |      297

 Cluster Management APIs

e-type": "sync"
},
{
"name": "eth2",
"ipv4-address": "192.168.1.254",
"ipv4-network-mask": "255.255.255.0",
"ipv4-mask-length": 24,
"ipv6-address": "",
"topology": "internal",
"topology-settings": {
"ip-address-behind-this-interface": "network defined by the interface ip and net
mask",
"interface-leads-to-dmz": false
},
"anti-spoofing": true,
"anti-spoofing-settings": {
"action": "prevent"
},
"security-zone": false,
"comments": "",
"color": "black",
"icon": "NetworkObjects/network",
"interface-type": "cluster"
}
]
},
"cluster-mode": "cluster-xl-ha",
"cluster-members": [
{
"name": "member1",
"sic-state": "initialized",
"sic-message": "Initialized but trust not established",
"ip-address": "172.23.5.1",
"interfaces": [
{
"name": "eth0",
"ipv4-address": "172.23.5.1",
"ipv4-network-mask": "255.255.255.0",
"ipv4-mask-length": 24,
"ipv6-address": "",
"ipv6-network-mask": "::",
"ipv6-mask-length": 0
},
{
"name": "eth1",
"ipv4-address": "1.1.1.1",
"ipv4-network-mask": "255.255.255.0",
"ipv4-mask-length": 24,
"ipv6-address": "",
"ipv6-network-mask": "::",
"ipv6-mask-length": 0
},
{
"name": "eth2",
"ipv4-address": "192.168.1.1",
"ipv4-network-mask": "255.255.255.0",
"ipv4-mask-length": 24,
"ipv6-address": "",
"ipv6-network-mask": "::",
"ipv6-mask-length": 0
}
]
},
{

ClusterXL R81 Administration Guide      |      298

 Cluster Management APIs

Example 10 - Viewing all existing cluster objects

API command:
Use this API to view all existing cluster objects.

show simple-clusters

Example 11 - Deleting an existing cluster object

API command:
Use this API to delete a specific cluster object.

delete simple-cluster

API example:

{
"name" : "cluster1"
}

Known Limitations
n These Cluster APIs support only subset of cluster operations.
n These Cluster APIs support only basic configuration of Software Blades (similar to "simple-
gateway" APIs - see the Check Point Management API Reference).
n These Cluster APIs support only ClusterXL High Availability, ClusterXL Load Sharing, and
CloudGuard OPSEC clusters.
n These Cluster APIs do not support the configuration of a Cluster Virtual IP address on a different
subnet than the IP addresses of the Cluster Members.
For such configuration, use SmartConsole.
n These Cluster APIs do not support VRRP Clusters (either on Gaia OS or IPSO OS).
n These Cluster APIs support a limited subset of interface settings.
To change interface settings such as Topology, Anti-Spoofing and Security Zone, you must replace
the interface.

ClusterXL R81 Administration Guide      |      299