Intelligent Disk Subsystems

ARCHITECTURE OF INTELLIGENT DISK SUBSYSTEMS

• In contrast to a file server, a disk subsystem can be

visualised as a hard disk server.
• Servers are connected to the connection port of the
disk subsystem using standard I/O techniques such as
Small Computer System Interface (SCSI), Fibre Channel
or Internet.
• SCSI (iSCSI) and can thus use the storage capacity that
the disk subsystem provides (Figure 2.1).
• The internal structure of the disk subsystem is
completely hidden from the server, which sees only the
hard disks that the disk subsystem provides to the
server.
• The connection ports are extended to the hard disks
of the disk subsystem by means of internal I/O
channels (Figure 2.2).
• In most disk subsystems there is a controller between
the connection ports and the hard disks. The
controller can significantly increase the data
availability and data access performance with the aid
of a so-called RAID procedure.
• some controllers realise the copying services instant
copy and remote mirroring and further additional
services.
• The controller uses a cache in an attempt to
accelerate read and write accesses to the server.
• Disk subsystems are available in all sizes. Small disk subsystems
have one to two connection ports for servers or storage
networks, six to eight hard disks and, depending on the disk
capacity, storage capacity of a few terabytes.
• Large disk subsystems have multiple ten connection ports for
servers and storage networks, redundant controllers and
multiple I/O channels. ( considerably larger number of servers
can access a subsystem through a connection over a storage
network.)
• Large disk subsystems can store up to a petabyte [ 10,00,000
GB] of data and, depending on the supplier, can weigh well over
a tonne [ 1000 Kg ].
• The dimensions of a large disk subsystem are comparable to
those of a wardrobe.
• Regardless of storage networks, most disk
subsystems have the advantage that free disk
space can be flexibly assigned to each server
connected to the disk subsystem (storage
pooling Fig : 2.3).
 HARD DISKS AND INTERNAL I/O CHANNELS

• The controller of the disk subsystem must

ultimately store all data on physical hard disks.
• Standard hard disks that range in size from
36GB to 1 TB are currently (2009) used for this
purpose.
• The maximum number of hard disks that can
be used is often limited, the size of the hard
disk used gives an indication of the maximum
capacity of the overall disk subsystem.
• When selecting the size of the internal physical hard
disks it is necessary to weigh the requirements of
maximum performance against those of the
maximum capacity of the overall system.
• With regard to performance ; it is often beneficial to
use smaller hard disks at the expense of the
maximum capacity: given the same capacity, if more
hard disks are available in a disk subsystem, the
data is distributed over several hard disks and thus
the overall load is spread over more arms and
read/write heads and usually over more I/O
Channels.
• For most applications, medium-sized hard
disks are sufficient. Only for applications with
extremely high performance requirements
should smaller hard disks be considered.
• consideration should be given to the fact that
more modern, larger hard disks generally have
shorter seek times and larger caches, so it is
necessary to carefully weigh up which hard
disks will offer the highest performance for a
certain load profile in each individual case.
• Standard I/O techniques such as SCSI, Fibre
Channel, increasingly Serial ATA (SATA) and Serial
Attached SCSI (SAS) and, still to a degree, Serial
Storage Architecture (SSA) are being used for
internal I/O channels between connection ports
and controller as well as between controller and
internal hard disks.
• Sometimes, however, proprietary – i.e.,
manufacturer-specific – I/O techniques are used.
Regardless of the I/O technology used, the I/O
channels can be designed with built-in redundancy
in order to increase the fault-tolerance of a disk
subsystem.
 Active
• In active cabling the individual physical hard disks are only
connected via one I/O channel (Figure 2.5, left). If this
access path fails, then it is no longer possible to access the
data.
Active/passive
• In active/passive cabling the individual hard disks are
connected via two I/O channels (Figure 2.5, right).
• In normal operation the controller communicates with the
hard disks via the first I/O channel and the second I/O
channel is not used.
• In the event of the failure of the first I/O channel, the disk
subsystem switches from the first to the second I/O
channel.
 Active/active (no load sharing)
• In this cabling method the controller uses both I/O channels in
normal operation (Figure 2.6, left).
• The hard disks are divided into two groups: in normal operation
the first group is addressed via the first I/O channel and the
second via the second I/O channel. If one I/O channel fails,
both groups are addressed via the other I/O channel.
Active/active (load sharing)
• In this approach all hard disks are addressed via both I/O
channels in normal operation (Figure 2.6, right). The controller
divides the load dynamically between the two I/O channels so
that the available hardware can be optimally utilised.
• If one I/O channel fails, then the communication goes through
the other channel only.
 JBOD: JUST A BUNCH OF DISKS
• If the disk subsystem has no internal controller, it is only an
enclosure full of disks (JBODs). In this instance, the hard disks are
permanently fitted into the enclosure and the connections for I/O
channels and power supply are taken outwards at a single point.
• A JBOD is simpler to manage than a few loose hard disks.
• Typical JBOD disk subsystems have space for 8 or 16 hard disks. A
connected server recognises all these hard disks as independent
disks. Therefore, 16 device addresses are required for a JBOD disk
subsystem incorporating 16 hard disks.
• JBOD has no controller (RAID controller) and intelligent
controller with additional services such as instant copy and
remote mirroring etc.,
 STORAGE VIRTUALISATION USING RAID
• A disk subsystem with a RAID controller offers
greater functional scope than a JBOD disk
subsystem.
• RAID was originally developed at a time when hard
disks were still very expensive and less reliable
than they are today.
• RAID was originally called ‘Redundant Array of
Inexpensive Disks’ or ‘Redundant Array of
Independent Disks’.
• Disk subsystems that support RAID are sometimes
also called RAID arrays.
• RAID has two main goals: Performance & Fault-
tolerance.
• To increase performance by striping and to
increase fault-tolerance by redundancy.
• Striping distributes the data over several hard
disks and thus distributes the load over more
hardware.
• Redundancy means that additional information
is stored so that the operation of the application
itself can continue in the event of the failure of a
hard disk.
• The bundle of physical hard disks brought together
by the RAID controller are also known as virtual
hard disks.
• A server that is connected to a RAID system sees
only the virtual hard disk; the fact that the RAID
controller actually distributes the data over several
physical hard disks is completely hidden to the
server (Figure 2.7).
• This is only visible to the administrator from outside.
• If a physical hard disk fails, its data can be
reconstructed from the hard disks that remain
intact.
• The defective hard disk can even be replaced by
a new one during operation; if a disk subsystem
has the appropriate hardware.
• The RAID controller reconstructs the data of the
exchanged hard disk. This process remains
hidden to the server apart from a possible
reduction in performance: the server can
continue to work uninterrupted on the virtual
hard disk.
 Hot spare disks
• Modern RAID controllers initiate this process
automatically. This requires the definition of so-called
hot spare disks (Figure 2.8).
• The hot spare disks are not used in normal operation. If
a disk fails, the RAID controller immediately begins to
copy the data of the remaining intact disk onto a hot
spare disk.
• Modern RAID controllers can manage a common pool of
hot spare disks for several virtual RAID disks.
• Hot spare disks can be defined for all RAID levels that
offer redundancy.
 DIFFERENT RAID LEVELS

• RAID has developed since its original definition

in 1987. Due to technical progress some RAID
levels are now practically meaningless, whilst
others have been modified or added at a later
date.
• This section introduces the RAID levels that are
currently the most significant in practice
 RAID 0: block-by-block striping
• RAID 0 distributes the data that the server writes to
the virtual hard disk onto one physical hard disk
after another block-by-block (block-by-block
striping). Figure 2.9 shows a RAID array with four
physical hard disks.
• RAID 0 increases the performance of the virtual hard
disk as follows: the individual hard disks can
exchange data with the RAID controller via the I/O
channel significantly more quickly than they can
write to or read from the rotating disk.
• Individual hard disks were able to achieve a
throughput of around 50MB/s. The four physical hard
disks achieve a total throughput of around 4 ×
50MB/s = 200MB/s.
• performance increases are only possible if the hard
disks are connected to different I/O paths so that the
load can be striped not only over several physical
hard disks, but also over several I/O paths.
• RAID 0 increases the performance of the virtual hard
disk, but not its fault-tolerance. If a physical hard disk
is lost, all the data on the virtual hard disk is lost.
• ‘RAID 0’ standing instead for ‘zero redundancy’.
 RAID 1: block-by-block mirroring
• in RAID 1 fault-tolerance is of primary importance
• The basic form of RAID 1 brings together two
physical hard disks to form a virtual hard disk by
mirroring the data on the two physical hard disks.
• If the server writes a block to the virtual
• hard disk, the RAID controller writes this block to
both physical hard disks (Figure 2.10).
• The individual copies are also called mirrors.
Normally, two or sometimes three copies of the
data are kept (three-way mirror).
• In a normal operation with pure RAID 1,
performance increases are only possible in read
operations. After all, when reading the data the
load can be divided between the two disks.
• This disadvantage can be disregarded for an
individual write operation, since the capacity of
the I/O channel is significantly higher than the
maximum write speed of the two hard disks put
together.
• The I/O channel is under twice the load, which
hinders other data traffic using the I/O channel at
the same time.
 RAID 0+1/RAID 10: striping and mirroring combined

• RAID 0 increase performance & RAID 1 is

fault-tolerance.
• Two RAID levels combine the ideas of RAID 0
and RAID 1. it would be nice to have both
performance and fault-tolerance. (RAID 0+1
and RAID 10 come into play)
• RAID 0+1 and RAID 10 each represent a two-
stage virtualisation hierarchy.
• Figure 2.11 shows the principle behind RAID
0+1 (mirrored stripes).
• In the example, eight physical hard disks are
used.
• The RAID controller initially brings together
each four physical hard disks to form a total of
two virtual hard disks that are only visible
within the RAID controller by means of RAID 0
(striping).
• In the second level, it consolidates these two
virtual hard disks into a single virtual hard disk
by means of RAID 1 (mirroring); only this
virtual hard disk is visible to the server.
RAID 10
• In RAID 10 (striped mirrors) the sequence of RAID 0
(striping) and RAID 1 (mirroring) is reversed in
relation to RAID 0+1 (mirrored stripes).
• In RAID 10 the RAID controller initially brings
together the physical hard disks in pairs by means of
RAID 1 (mirroring) to form a total of four virtual hard
disks that are only visible within the RAID controller.
• In the second stage, the RAID controller consolidates
these four virtual hard disks into a virtual hard disk by
means of RAID 0 (striping). Here too, only this last
virtual hard disk is visible to the server.
 which of the two RAID levels, RAID 0+1 or RAID 10,
is preferable?

• In the example relating to RAID 0+1 the

failure of a physical hard disk is thus
equivalent to the effective failure of four
physical hard disks.
• If one of the other four physical hard disks is
lost, then the data is lost.
• RAID 10 thus has a significantly higher fault-
tolerance than RAID 0+1.
 RAID 4 and RAID 5: parity instead of mirroring
• The idea of RAID 4 and RAID 5 [Stripped Parity] is to
replace all mirror disks of RAID 10 with a single parity
hard disk.
• the principle of RAID 4 based upon five physical hard
disks. The server again writes the blocks A, B, C, D, E,
etc. to the virtual hard disk sequentially.
• The RAID controller stripes the data blocks over the first
four physical hard disks. Instead of mirroring all data
onto the further four physical hard disks, as in RAID 10,
the RAID controller calculates a parity block for every
four blocks and writes this onto the fifth physical hard
disk.
• From a mathematical point of view the parity
block is calculated with the aid of the
• logical XOR operator (Exclusive OR).
• In the example from Figure 2.15, for example,
the equation
• P(ABCD) = A XOR B XOR C XOR D applies.
• Changing a data block changes the value of the
associated parity block.
• This means that each write operation to the virtual
hard disk requires
(1) the physical writing of the data block,
(2) the recalculation of the parity block and
(3) the physical writing of the newly calculated parity
block.

This extra cost for write operations in RAID 4 and RAID

5 is called the write penalty of RAID 4 or the write
penalty of RAID 5.
• The cost for the recalculation of the parity block is relatively low due to the mathematical
properties of the XOR [⊕] operator.
• If the block A is overwritten by block Ãand is the difference between the old and new
data block, then

Δ = A ⊕ Ã.
The new parity block ˜P can now simply be calculated from
the old parity block P and , i.e. p̂ = P ⊕ Δ

• Therefore, if P (ABCD) is the parity block for the data blocks A, B, C and D, then after
the data block A has been changed, the new parity block can be calculated without
knowing the remaining blocks B, C and D.
• However, the old block A must be read in before overwriting the physical hard disk
in the controller, so that this can calculate the difference .
•Then it uses the ⊕ operation to calculate the difference between the old and the new parity
block, i.e. Δ = D XOR D˜ , and from this the new

parity block p̂ (ABCD) by means of

• p̂ (ABCD) = p(ABCD) ⊕ Δ .

Therefore it is not necessary to read in all four associated data blocks to recalculate the parity block.
To
 RAID 2 and RAID 3
• In RAID 2 the Hamming code is used, so that
redundant information is stored in addition to the
actual data. This additional data permits the
recognition of read errors and to some degree also
makes it possible to correct them.
• RAID 3 stores parity data and it distributes the data
of a block amongst all the disks of the RAID 3
system so that, in contrast to RAID 4 or RAID 5, all
disks are involved in every read or write access.
A comparison of the RAID levels
 CACHING: ACCELERATION OF HARD DISK ACCESS

• In all fields of computer systems, caches are used

to speed up slow operations by operating them
from the cache.
• Specifically in the field of disk subsystems, caches
are designed to accelerate write and read accesses
to physical hard disks.
• two types of cache:
• (1) Cache on the hard disk and
(2) Cache in the RAID controller.
• The cache in the RAID controller is subdivided into
write cache and read cache
 Cache on the hard disk
• This is necessary because the transfer rate of the
I/O channel to the disk controller is significantly
higher than the speed at which the disk controller
can write to or read from the physical hard disk.
• The disk controller can thus write the block to the
physical hard disk in its own time whilst the I/O
channel can be used for data traffic to the other
hard disks.
• While reading the hard disk controller transfers the
block from its cache to the RAID controller or to
the server at the higher data rate of the I/O
channel.
 Instant copies
• Instant copies can virtually copy data sets of
several terabytes within a disk subsystem in a
few seconds. Virtual copying means that disk
subsystems fool the attached servers into
believing that they are capable of copying
such large data quantities in such a short
space of time.
• The actual copying process takes significantly
longer.
 Remote mirroring
• Remote mirroring offers protection against
such catastrophes. Modern disk subsystems
can now mirror their data, or part of their
data, independently to a second disk
subsystem, which is a long way away.
 High availability with remote mirroring:
(1) The application server stores its data on a local
disk subsystem.
(2) The disk subsystem saves the data to several
physical drives by means of RAID.
(3) The local disk subsystem uses remote mirroring to
mirror the data onto a second disk subsystem located
in the backup data centre.
(4) Users use the application via the LAN.
(5) The stand-by server in the backup data centre is
used as a test system. The test data is located on a
further disk subsystem.
(6) If the first disk subsystem fails, the application is
started up on the stand-by server using the data of
the second disk subsystem.
(7) Users use the application via the WAN.
In synchronous remote mirroring a disk subsystem does not acknowledge
write operations until it has saved a block itself and received write
confirmation from the second disk subsystem.
In asynchronous remote mirroring one disk subsystem
acknowledges a write operation as soon as it has saved the
block itself.
 The combination of synchronous and asynchronous
remote mirroring means that rapid response times can
be achieved in combination with mirroring over long
distances.
 LUN masking

Chaos: Each server works to Each server sees all hard

its own virtual hard disk. disks. A configuration
Without LUN masking error on server 1 can
destroy the data on the
other two servers. The
data is thus poorly
protected.
Order: each server works to its own virtual hard disk. With LUN
masking, each
server sees only its own hard disks. A configuration error on server 1
can no longer destroy the data of the two other servers. The data is
now protected.
 Port-based LUN masking & Server-based LUN masking

• Port-based LUN masking is the ‘poor man’s LUN

masking’, it is found primarily in low-end disk
subsystems. In port-based LUN masking the filter only
works using the granularity of a port. This means that all
servers connected to the disk subsystem via the same
port see the same disks.
• Server-based LUN masking offers more flexibility. In this
approach every server sees only the hard disks assigned
to it, regardless of which port it is connected via or
which other servers are connected via the same port.
 AVAILABILITY OF DISK SUBSYSTEMS

• The following list describes the individual

measures that can be taken to increase the
availability of data:
The data is distributed over several hard disks
using RAID processes and supplemented by
further data for error correction. After the
failure of a physical hard disk, the data of the
defective hard disk can be reconstructed from
the remaining data and the additional data.
Self-diagnosis functions in the disk controller
continuously monitor the rate of bit errors and
the physical variables (e.g., temperature,
spindle vibration). In the event of an increase
in the error rate, hard disks can be replaced
before data is lost.
 Each internal physical hard disk can be
connected to the controller via two internal
I/O channels. If one of the two channels fails,
the other can still be used.
 Other auxiliary components such as power
supplies, batteries and fans can often be
duplicated so that the failure of one of the
components is unimportant. When connecting
the power supply it should be ensured that the
various power cables are at least connected
through various fuses.
 Server and disk subsystem are connected
together via several I/O channels. If one of the
channels fails, the remaining ones can still be
used.
Instant copies can be used to protect against
logical errors. For example, it would be possible
to create an instant copy of a database every
hour. If a table is ‘accidentally’ deleted, then
the database could revert to the last instant
copy in which th
Remote mirroring protects against physical
damage. If, for whatever reason, the original
data can no longer be accessed, operation can
continue using the data copy that was
generated using remote mirroring. e database
is still complete.
o Consistency groups and write-order consistency
synchronise the copying of multiple virtual hard
disks. This means that instant copy and remote
mirroring can even guarantee the consistency of
the copies if the data spans multiple virtual hard
disks or even multiple disk subsystems.
LUN masking limits the visibility of virtual hard
disks. This prevents data being changed or
deleted unintentionally by other servers.
Thank U