Module 13

Synchronous Replication of Volumes

 By the end of the module, you should be able to:

• Identify and explain the differences between Asynchronous and Synchronous Replication
• Define the EqualLogic requirements and implementation of Synchronous Replication (SyncRep)

• Configure a Group to support SyncRep

• Configure a volume for SyncRep

• Demonstrate the different states that a SyncRep volume can be in

• Explain the difference between switching over and failing over a SyncRep volume

• Identify and describe the differences between implementing SyncRep with (a) single member pool(s) and
(a) multi-member pool(s)

Dell - Restricted - Confidential

2 Dell - Restricted - Confidential
Internal Use - Confidential
 Synchronous Replication

PS Storage Replication
Methods

Dell - Restricted - Confidential

3 Dell - Restricted - Confidential
Internal Use - Confidential
 EqualLogic Replication Methods

• PS Series firmware provides the following methods for automatically replicating block volumes to
provide protection against accidental data loss:
• Traditional replication (referred to as Replication or Auto-Replication) is a point-in-time process that is conducted
between two groups, often in geographically diverse locations. Replication provides protection against a regional disaster
such as an earthquake or hurricane.
• Synchronous replication (also known as SyncRep) is a real-time process that simultaneously writes volume data across
two different pools within the same PS Series group. This method is useful for maintaining two copies of a volume’s data
in the same data center, or dispersed to two different facilities on the same campus or in the same metropolitan area.

• Note: You cannot enable synchronous replication on a volume for which traditional replication is
configured, and you cannot enable traditional replication on a volume for which synchronous replication
is configured.

Dell - Restricted - Confidential

4 Dell - Restricted - Confidential
Internal Use - Confidential
 What is SyncRep?

• SyncRep is a mechanism that will allow for a volume to be synchronously replicated within a Group
• Distance limited
• Requires a Group with at least two pools (minimum of two arrays)
• Each pool contains a copy of the volume

• Every write to the volume is acknowledged by both pools before an acknowledgement is sent back to the
initiator host

• SyncRep provides near-zero recovery time objectives (RTOs) and real time crash-consistent Recovery
Point Objective (RPOs)

• Highly desirable for critical applications that require volumes with high availability

Dell - Restricted - Confidential

5 Dell - Restricted - Confidential
Internal Use - Confidential
 Comparing SyncRep with Traditional (Async) Replication

Dell - Restricted - Confidential

6 Dell - Restricted - Confidential
Internal Use - Confidential
 Synchronous Replication

Synchronous Replication
(SyncRep)

Dell - Restricted - Confidential

7 Dell - Restricted - Confidential
Internal Use - Confidential
 SyncRep – Vocabulary & Terms

• Pool Roles
• SyncActive Pool
• The SyncActive Pool is the pool where the active image resides.
• SyncAlternate Pool
• This pool is where the alternate image of the volume reside
• A pool can contain both Active and Alternate volume images, and the description of the pool is directly related to the volume’s state

• Volume States
• In sync
• The volume images from each pool are identical
• Paused
• Only the SyncActive Pool will acknowledge writes. If any writes come in while paused, the changes to the volume are tracked until SyncRep is resumed
• Out-of-sync
• The SyncAlternate volume image does not contain the same data as the SyncActive image
• Can occur if paused, if pool (or members in pool) becomes unavailable, if pool is out of free space, if snapshot reserve space for the volume is full

Dell - Restricted - Confidential

8 Dell - Restricted - Confidential
Internal Use - Confidential
 About SyncRep

• Introduced with Firmware version 6.0

• Requirements
• Requires Two Pools – one for each SyncRep volume (Sync Active/ Sync Alternate)
• Pools must reside on the same subnet
• Requires fast & reliable connectivity
• Tolerance for latency is dependent upon the application using the volume
• Cannot be configured for Asynchronous Replication
• Strongly recommend disabling UNMAP operations on SyncRep volumes
• Maximum of 32 volumes can be configured for SyncRep (4 for PS4x00)
• Must have a minimum of 5% Snapshot Reserve

• Snapshots supported
• Snapshots only occur on SyncActive Pool
• Snapshots are not mirrored to SyncAlternate Pool

Dell - Restricted - Confidential

9 Dell - Restricted - Confidential
Internal Use - Confidential
 Replication Models with SyncRep

• Depending on the number of pools and

members, you can design the Group’s
pools to support a matrix/mesh model
for SyncRep
• One Way replication, Reciprocal
replication can be established
between pools

Dell - Restricted - Confidential

10 Dell - Restricted - Confidential
Internal Use - Confidential
 Synchronous Replication

How SyncRep Works

Dell - Restricted - Confidential

11 Dell - Restricted - Confidential
Internal Use - Confidential
 How SyncRep Works – In Sync

• Normal operation
• With every SCSI WRITE request, the
target sends back an acknowledgement
• SyncRep Configured Volume
• Appears as a single volume in Group
Manager and to Host
• SyncRep manages Active and Alternate page
tables / volumes
• Both volumes have the same IQN
• SyncRep enabled volume will only send back
an acknowledgement once both copies from
each pool have acknowledged the write
• Either volume can be made the Active
volume at any time while volume is In Sync
Dell - Restricted - Confidential
12 Dell - Restricted - Confidential
Internal Use - Confidential
 How SyncRep Works – Paused

• When SyncRep is in a Paused state, it will

not send writes to the SyncAlternate
• It's possible for a SyncRep volume to be in
both a Paused and In Sync state
• Its possible to Switch to SyncAlternate
image while paused as long as the volume is
In Sync
• When a write is received, to the SyncActive
while in a Paused state, the SyncRep state
becomes Out of Sync

Dell - Restricted - Confidential

13 Dell - Restricted - Confidential
Internal Use - Confidential
 SyncActive & SyncAlternate Relationship with Volume

• Why Change Pool Relationship?

• SyncActive member or pool failure (failover)
• Planned Maintenance on SyncActive members (switchover)

• Switchover
• Only available when volume images are In Sync
• Typically used for planned maintenance
• Can only while paused as long as images are In Sync

• Failover
• Replaces Switchover as an option when volumes images are Out of Sync
• This is the choice when the volume is out of sync and a fault has already occurred
• Used when some data loss is acceptable over unavailable data

• Both Switchover and Failover force the initiator to reconnect to the volume through an Async Logout
• Allows for the use of the same IP and IQN string to connect to volume in alternate pool

Dell - Restricted - Confidential

14 Dell - Restricted - Confidential
Internal Use - Confidential
 How SyncRep Works When Alternate Paused/Offline & Out of Sync

• As soon as a write is received to the

SyncActive while in a Paused state, the
SyncRep state becomes Out of Sync
• A snapshot is taken and all writes that come
in while the SyncAlternate is down are
tracked
• When volume is unpaused, the differences
between the snapshot and production image
will be sent to the Out of Sync image
• Normal I/O will still be tracked until all of
the tracked changes are synchronized with
Alternate image
Dell - Restricted - Confidential
15 Dell - Restricted - Confidential
Internal Use - Confidential
 Synchronous Replication

Configuring a Volume for

SyncRep

Dell - Restricted - Confidential

16 Dell - Restricted - Confidential
Internal Use - Confidential
 Configuring SyncRep on a Volume – Group Manager GUI

• Select the volume and choose

Configure SyncRep
• Go through the Two step Configure
SyncRep Wizard
• Specify the SyncAlternate Pool
• Review Summary and Finish
• Volume Icon will change
• Status Page will indicate SyncRep
specific space consumption

Dell - Restricted - Confidential

17 Dell - Restricted - Confidential
Internal Use - Confidential
 Configuring SyncRep on a Volume – Group Manager CLI

• You’ll need to know

• Volume name
• Name of pool you will designate
SyncAlternate
• Command Syntax
• volume select <VOLUMENAME>
sync-rep enable
<SyncAlt_POOLNAME>

Dell - Restricted - Confidential

18 Dell - Restricted - Confidential
Internal Use - Confidential
 Group Manager Volume – SyncRep Activities & Status

Dell - Restricted - Confidential

19 Dell - Restricted - Confidential
Internal Use - Confidential
 Synchronous Replication

Sync Active and Sync

Alternate Behavior

Dell - Restricted - Confidential

20 Dell - Restricted - Confidential
Internal Use - Confidential
 Switching Pool Relationship – In Sync

• Switching to SyncAlternate
• Only available when Volume Images are
In Sync
• Change the role of the pools
• Switching to SyncAlternate will cause
iSCSI initiators to be to be redirected to
the new SyncActive volume
• The switch should complete in a second
(or less) in a low latency network

Dell - Restricted - Confidential

21 Dell - Restricted - Confidential
Internal Use - Confidential
 When the SyncAlternate Goes Offline (In or Out of Sync)

• A system snapshot is automatically taken of the Active Image when the SyncRep
volumes go Out of Sync
• When a write comes into the active image, SyncRep becomes Out of Sync, and
Switchover to SyncAlternate is no longer allowed
• Failover to SyncAlternate will appear when the relationship becomes Out of Sync
• You should not attempt to failover to an Out of Sync Offline volume unless data loss is
acceptable

Dell - Restricted - Confidential

22 Dell - Restricted - Confidential
Internal Use - Confidential
 When the SyncActive Goes Offline While In Sync

• Failover is not automatic

• Should only be used in extraordinary circumstances
• Volume will become unavailable until failover occurs
• Because SyncRep state is In Sync, no data will be lost
• Steps to recover
• Take SyncRep volume offline
• Failover to SyncAlternate
• Bring SyncRep volume online
• Host login to SyncRep volume

• Restore down Pool to resume normal operation

Dell - Restricted - Confidential
23 Dell - Restricted - Confidential
Internal Use - Confidential
 When the SyncActive Goes Offline While Out of Sync

• Data will be lost and Volume Availability will be down

• Failover is not automatic – Host connectivity is lost
• Steps to recover
• Take SyncRep volume offline
• Failover to SyncAlternate – Failover is to older data
• Bring SyncRep volume online
• Host login to SyncRep volume
• Data from time images went Out of Sync until SyncActive pool went offline is lost

Dell - Restricted - Confidential

24 Dell - Restricted - Confidential
Internal Use - Confidential
 Failover to SyncAlternate Procedure

• Failover should only be used in extraordinary circumstances

• Check SyncRep status
• If Out of Sync data will be lost
• If In Sync, impact is to availability only
• Take the SyncRep Volume Offline
• If you do not, you will encounter an error if you try to Failover
• Failover to SyncAlternate Volume
• If Out of Sync, data will be lost
• If at least one member of the faulted SyncActive pool is
available, you should be able to proceed
• If all members of the SyncActive pool are unavailable, you will
be prompted to Disconnect the SyncActive before proceeding
• Bring the Volume Online & reconnect host’s iSCSI Login
Dell - Restricted - Confidential
25 Dell - Restricted - Confidential
Internal Use - Confidential
 SyncRep and Snapshot Behavior

• Creating a snapshot of a volume will create a snapshot in the SyncActive pool only
• As with snapshots of volumes for which synchronous replication is not enabled, you can access the data in
the snapshot by setting the snapshot online or cloning it regardless of what pool it is in.

• To restore a volume from snapshot the snapshot must be in the SyncActive pool

• Switchover / Failover moves where a schedule creates a snapshot to the new SyncActive pool, but
existing snapshots remain in the pool they were created in
• Temporary snapshots are created in both pools to ensure changes to the volume are tracked if the volume
is out of sync.

Dell - Restricted - Confidential

26 Dell - Restricted - Confidential
Internal Use - Confidential
 SyncRep Snapshots

• Snapshots can exist and be accessible in either pool but are only created in the pool
that is SyncActive Pool at the time of creation
• Restoring a Snapshot based in the SyncAlternate will prompt you to Switch to
SyncAlternate before the restore occurs

Dell - Restricted - Confidential

27 Dell - Restricted - Confidential
Internal Use - Confidential
 Collections and SyncRep
• Before you Enable SyncRep for Collection all volumes must:
• Have SyncRep already enabled
• Must have a SyncRep status of In Sync
• Must use the same two storage pools
• Must not belong to another SyncRep-enabled collection
• Cannot be moving from one storage pool to another
• Collections can contain no more than 8 volumes
• After you Enable SyncRep for the Collection, all SyncRep activities should be performed at the
Collection level
• Allows multiple volumes to act as one volume for SyncRep functionality
• Allows all volumes to be Paused at the same time
• Allows all volumes to be Switchover or Failover at the same time
• Guarantees that all volumes have the same crash consistent point in time
• You can pause individual volumes
• Reasons to create Volume collections
• Allows ability to create crash consistent snapshots across multiple volumes
• Do Not use as a substitute for Volume Folders
Dell - Restricted - Confidential
28 Dell - Restricted - Confidential
Internal Use - Confidential
 Synchronous Replication

Troubleshooting SyncRep

Dell - Restricted - Confidential

29 Dell - Restricted - Confidential
Internal Use - Confidential
 SyncRep Troubleshooting Notes

• Failover situations
• If the SyncActive volume becomes unavailable while the volume is in sync, you can safely failover to the SyncAlternate
volume. Although host access to the volume is disrupted during the switch, no initiator changes are required
• If the SyncActive volume becomes unavailable while the volume is out of sync, any changes written to the SyncActive
volume that have not yet been replicated to the SyncAlternate pool will be lost and cannot be recovered. Failovers should
only be performed under extraordinary circumstance

• Changes cannot be tracked in a snapshot if there is not enough snapshot reserve space

• SyncRep must be disabled before a volume can be saved in the volume recovery bin

Dell - Restricted - Confidential

30 Dell - Restricted - Confidential
Internal Use - Confidential
 SyncRep Networking Considerations

• Follow same networking best practices for standard group setup

• Utilize a “Flat” network infrastructure for SyncRep
• Same subnet, non-routed
• Compression, de-duplication and encryption all add latency and are not recommended

• Utilize the same switch technologies for the SyncActive and SyncAlternate Pools (Force10 is preferred)
• When using multiple switches ensure only a single hop between switches

• Maintain short distance between Pools for acceptable level of latency tolerance.
• Latency is determined by 2 factors – Distance between pools and interaction between TCP reliability and congestion
control protocols.
• To minimize application performance impact it is recommended having the source and destination PS Pools located
within 1km in distance

Dell - Restricted - Confidential

31 Dell - Restricted - Confidential
Internal Use - Confidential
 Synchronous Replication

Module Review and Lab

Dell - Restricted - Confidential

32 Dell - Restricted - Confidential
Internal Use - Confidential
 End of Module Review

• In this module we have reviewed and discussed the following:

• Identified and explained the differences between Asynchronous and Synchronous Replication
• Defined the EqualLogic requirements and implementation of Synchronous Replication (SyncRep)
• Configured a Group to support SyncRep
• Configured a volume for SyncRep
• Demonstrated the different SyncRep volume states
• Explained the difference between switching over and failing over a SyncRep volume
• Identified and described the differences between implementing SyncRep with (a) single member pool(s) and (a) multi-
member pool(s)

Dell - Restricted - Confidential

33 Dell - Restricted - Confidential
Internal Use - Confidential
 Synchronous Replication Lab

• Please refer to the lab manual for instructions.

Dell - Restricted - Confidential

34 Dell - Restricted - Confidential
Internal Use - Confidential
Module 14
Collecting Diagnostics Data
 By the end of the module, you should be able to:

• Describe, compare and/or demonstrate the differences between running diagnostics from within the Group
Manager GUI and the CLI
• Identify the 15 different diagnostics sections that are collected when Diagnostics are run on an array

• Describe what is meant by ‘abbreviated’ diagnostics and when they are useful

• Identify the different diagnostic parameters and switches that you may need to employ when running diagnostics

• Identify the output files that the diagnostics script generates

• Identify the different methods to extract the diagnostic output from a member or group of members

• Determine how to verify a hard disk failure using Group Manager

• Determine how to verify a power supply failure using Group Manager

Dell - Restricted - Confidential

36 Dell - Restricted - Confidential
Internal Use - Confidential
 Collecting Diagnostics
Data

Collecting PS Series
Array Diagnostic Data

Dell - Restricted - Confidential

37 Dell - Restricted - Confidential
Internal Use - Confidential
 Dell PS Series Array Diagnostics

• SAN Assist is preferred method for collecting Group/Array Diagnostics for customers
• Running Dell EqualLogic Array Diagnostics
• GUI
• Easiest method to execute for customer
• Can run across multiple members at the same time
• Cannot pass parameters to the ‘under-the-hood’ diagnostic script
• CLI
• Must connect directly to the member (not the Group IP)
• Must run manually on every member
• Can run any subset of the diagnostics collectors (sometimes required by customers)
• Many parameters available to CLI, including abbreviated output

Dell - Restricted - Confidential

38 Dell - Restricted - Confidential
Internal Use - Confidential
 Collecting Diagnostics with Support Assist

• Customer oriented for automated e-support

• Runs automatically when customer installs
SupportAssist
• Manual option available
• Accessible from SAN Headquarters via Support Assist
menu in tool bar
• Customer can ‘decrypt’ output to allay any security
concerns
• If calling support, will use array diagnostic script via Group
Manager GUI or CLI

Dell - Restricted - Confidential

39 Dell - Restricted - Confidential
Internal Use - Confidential
 Array Diagnostics Data Collected

• When diagnostics is run, multiple encrypted files (seg_#.dgo) are generated and stored
on the array (grpadmin’s / ftp directory)
• Full Diagnostics collects 15 groups (or sections) of information:

Dell - Restricted - Confidential

40 Dell - Restricted - Confidential
Internal Use - Confidential
 Manually Running Diagnostics

• Passing parameters to diag script is only available with the CLI

• Required if a customer disallows SNMP collection (diag -x14)
• seg_#.dgo encrypted files generated and accessible via integrated FTP server
• Ability to capture through terminal console if necessary
• Ability to allow/disallow sending email with results and/or run abbreviated diags
• Use -w when possible, as snmpwalk is quicker than snmpget

Dell - Restricted - Confidential

41 Dell - Restricted - Confidential
Internal Use - Confidential
 Manually Running Array Diagnostics - CLI

• Login directly to the member as grpadmin (or any Group Administrator account)
• Determine if any parameters need to be passed to script
• To the management port if enabled, otherwise any eth port or via the active CM’s serial port
• ALWAYS run diagnostics on the Group Lead in addition to problem /suspect arrays
• If a small group (e.g. 4 arrays), run diags on all members in the group
• Execute script with any necessary parameters
• Enter diag -h for command syntax & help
• <CTRL>+C can break out of the script

Dell - Restricted - Confidential

42 Dell - Restricted - Confidential
Internal Use - Confidential
 How to Manually Run Array Diagnostics - GUI

• Login as a Group Administrator

• Open the Tools panel and click on
Diagnostic reports
• Complete the Diagnostics Wizard
• Always on the Group Lead
• All suspect arrays
• All arrays if small group
• Check status of diags through
Operations Panel

Dell - Restricted - Confidential

43 Dell - Restricted - Confidential
Internal Use - Confidential
 Retrieving Diagnostic Files From The Array(s)

• Requirements
• Host that can communicate over
management network to the array
• Over iSCSI SAN if management port
not configured due to mixed traffic
• FTP Client
• FileZilla, WinSCP (FTP protocol)
• Command prompt

Dell - Restricted - Confidential

44 Dell - Restricted - Confidential
Internal Use - Confidential
 .DGO files – What Do I Do With Encrypted .DGO Files?

• .DGO Files must be uploaded to the Dell EqualLogic

Diagnostic Server (a.k.a., “Diag Server”)
• MUST be placed in correct folder to be useful and
retain historical reference
• When not properly uploaded, customer may experience an
issue and support personnel not seeing or finding it
• Diag Server is a Linux host that is the central repository
for diagnostic data that is used by support for analysis
• File structure is specific
• Many scripts are used to assist in analyzing
“unpacked” .DGO files

Dell - Restricted - Confidential

45 Dell - Restricted - Confidential
Internal Use - Confidential
 Collecting Diagnostics
Data

Diagnosing Hard Drive

and Power Failure Faults

Dell - Restricted - Confidential

46 Dell - Restricted - Confidential
Internal Use - Confidential
 Diagnosing Hard Disk and Power Supply Failures

• Hard Disk Failures

• Verify/validate using Group Event Log through SAN Headquarters or Group Manager
• Check disk firmware revisions
• Do NOT reseat disks, replace them

• Power Supply Failures

• Validate environment/cables
• Verify/validate fault using Group Event Log through SAN Headquarters or Group Manager

Dell - Restricted - Confidential

47 Dell - Restricted - Confidential
Internal Use - Confidential
 When a Disk Fails or Has Tripped SMART
• Tripped SMART on array running firmware 6.01 or later
• Drive will begin copy-to-spare (pre-emptive mirroring) prior to fault
• If drive fails during process, will shift to normal rebuild process

• Fails with a spare drive

• Spare replaces failed disk and begins RAID rebuild
• RAID-6 and RAID-5 have 1 spare by default
• RAID-10 and RAID-50 members have 2 spares by default

• Fails with no spare drive available

• Data continues to be available, but the set is degraded.

• Multiple Disk Failure (MDF)

• In a degraded RAID 10, RAID 5, or RAID 50 set, the member will be set offline; any volumes and snapshots residing on member are set offline
• In a degraded RAID 6 set, continues in degraded state with further performance degradation. After both drives are reconstructed, performance returns to normal.

• When a failed drive is replaced:

• If a spare drive was used: Data has already been reconstructed on the spare drive, so the new drive becomes a spare. EqualLogic does not have a specific slot
reserved for spare.
• If a set is degraded: Data is reconstructed on the new drive and after reconstruction, performance returns to normal. (NOTE: CLI commands exist that allow a
customer to configure RAID Policy with no spares)

Dell - Restricted - Confidential

48 Dell - Restricted - Confidential
Internal Use - Confidential
 Determine Status of a Hard Disk

• Check the Event Log (SAN Headquarters or Group Manager)

• Use Group Manager to review Member information
• Abbreviated diag (diag -ad) is available, but does not replace Diagnostics

Dell - Restricted - Confidential

49 Dell - Restricted - Confidential
Internal Use - Confidential
 Diagnosing Power Supply Faults

• Cooling Fans are a part of the Power

Supply
• Check the Event Log (SAN Headquarters
or Group Manager)
• Use Group Manager to review Member
information
• CLI: select member show enclosure
• GUI: Select Member -> Status Tab (Rear
view) and -> Enclosure Tab
• Abbreviated diag (diag -ap) is available, but
does not replace Diags
Dell - Restricted - Confidential
50 Dell - Restricted - Confidential
Internal Use - Confidential
 Collecting Diagnostics
Data

Module Review and Lab

Dell - Restricted - Confidential

51 Dell - Restricted - Confidential
Internal Use - Confidential
 End of Module Review

• In this module we have reviewed and discussed the following:

• The differences between running diagnostics from within the Group Manager GUI and the CLI
• The 15 different diagnostics sections that are collected when Diagnostics are run on an array
• What is meant by ‘abbreviated’ diagnostics and when they are useful
• Why you might need to run only partial diagnostics
• The different diagnostic parameters and when running diagnostics
• The output files the diagnostics script generates
• The different methods to extract the diagnostic output from a member or group of members
• How to verify a hard disk failure using Group Manager
• How to verify a power supply failure using Group Manager

Dell - Restricted - Confidential

52 Dell - Restricted - Confidential
Internal Use - Confidential
 Collecting Diagnostics Data Lab

• Please refer to the lab manual for instructions.

Dell - Restricted - Confidential

53 Dell - Restricted - Confidential
Internal Use - Confidential