IT Infrastructure

Architecture
Infrastructure Building Blocks
and Concepts

Storage – Part 2
(chapter 10)
Network Attached Storage (NAS)

• A NAS, also known as a File Server, is a network

device that provides a shared file system to
operating systems over a standard TCP/IP
network
 NFS (UNIX and Linux)
 SMB/CIFS (Windows)

• A NAS is often an appliance that implements the

file services and holds the disks on which data is
stored
• A NAS appliance could also use external disk
storage provided by a SAN
• Can provide snapshot and clone technology at a
file level, enabling features like “un-erasing”
deleted files by end users
Network Attached Storage (NAS)

• The difference between a SAN and NAS:

 SAN:
 Offers disk blocks (unformatted disks called LUNs) that can be used by only one server
 Uses iSCSI, Fibre Channel or FCoE as the communication layer
 NAS:
 Offers a shared filesystem to store files that can be used by multiple servers
 Connects to for instance to an LDAP or Active Directory service in order to set file and/or
folder permissions
 Uses SMB/CIFS or NFS over TCP/IP as the communication layer
Network Attached Storage (NAS)

• A clustered NAS is a NAS that uses a distributed file system running

simultaneously on multiple servers
 Distributes data and metadata across storage devices
 Still provides unified access to the files from any of the cluster nodes, unrelated to the
actual location of the data

• File shares can also be provided by public cloud providers

 AWS offers File Gateways
 Azure has Storage Accounts
 GCP has Filestore
Object Storage

• Object storage manages data as objects

 An object is defined as a file with its metadata, and a globally unique identifier called
the object ID

• Object storage stores and retrieves data using a REST API calls over HTTP,
is served by a webserver, and is designed to be highly scalable
• All large public cloud providers offer object storage services
 AWS has S3
 Azure has Blob Storage as part of storage accounts
 GCP provides Object Storage

• These services are massively scalable and data is stored in multiple

locations geographically to ensure no data will be lost in case of a disaster
Object Storage

• Data in object storage can’t be modified

 If a file is modified, the original file is deleted, and a new file is created

• Object storage is unsuitable for frequently changing data

• It is a good fit for data that doesn't change much, like office documents,
backups, archives, video and audio files, and virtual machine images
• Some systems emulate a file system using object storage
 For instance, Amazon’s S3FS creates a virtual filesystem, based on S3 object storage,
that can be mounted to an operating system in the traditional way, however, with
significant performance degradation
 A much better solution is to use object storage with applications designed for it
Software Defined Storage

• Software Defined Storage (SDS)

abstracts data and storage
capabilities (also known as the
control plane) from the
underlying physical storage
systems (the data plane)
Software Defined Storage

• SDS virtualizes all physical storage into one large shared storage pool
 Data can be stored in a variety of storage systems while being presented and managed
as one storage pool to the servers consuming the storage

• Storage can be implemented as software running on commodity x86-

based servers with direct attached disks
• Physical storage can also be a SAN, a NAS, or an Object storage system
• The storage offers from all major cloud providers is based on SDS
 All storage options can be configured using APIs, CLI, and a GUI in a web portal
Software Defined Storage

• From the shared storage pool, software provides data services like:
 Deduplication
 Compression
 Caching
 Snapshotting
 Cloning
 Replication
 Tiering
Software Defined Storage

• SDS provides servers with virtualized data storage pools

 With the required performance, availability and security
 Delivered as block, file, or object storage
 Based on policies

• Example:
 A newly deployed database server can invoke an SDS policy that mounts storage
configured to have its data striped across a number of disks, creates a daily snapshot,
and has data stored on tier 1 disks

• APIs can be used to provision storage pools and set the availability,
security and performance levels of the virtualized storage
• Using APIs, storage consumers can monitor and manage their own storage
consumption
Storage availability
Redundancy and data replication
• To increase availability in a SAN, components like HBAs and switches can be installed
redundantly
• Using multiple paths between HBAs and SAN switches, failover can be instantiated
automatically when a failure occurs
• Multiple storage systems can be used. Using replication, changed disk blocks from the
primary storage system are continuously sent to the secondary storage system, where
they are stored as well
Redundancy and data replication

• Synchronous replication:
 Each write to the active storage system and the replication to the passive storage
system must be completed before the write is confirmed to the operating system
 Ensures data on both storage systems is synchronized at all times and data is never lost
 When the physical cable length between the two storage systems is more than 100 km,
latency times get too long, slowing down applications, that have to wait for the write
on the secondary storage system to finish
 Risk: a failing connection between both storage systems a write is never finished, as the
data cannot be replicated. This effectively leads to downtime of the primary storage
system
Redundancy and data replication

• Asynchronous replication:
 After data has been written to the primary storage system, the write is immediately
committed to the operating system, without having to wait for the secondary storage
array to finish its writes as well
 Asynchronous replication does not have the latency impact that synchronous
replication has
 Disadvantage: potential data loss when the primary storage system fails before the data
has been written to the secondary storage system
Backup and recovery

• Replication does not protect against data erasure or data integrity

problems due to a software bug. If this happens, the deleted or altered
data will also be replicated to the secondary storage system
 Backups are needed to protect from this

• Backups are copies of data, used to restore data to a previous state in case
of data loss, data corruption or a disaster recovery situation
• Backups are always a last resort, only used if everything else fails, to save
your organization in case of a disaster
• A well-designed system should have options to repair incorrect data from
within the system or by using systems management tools (like database
tools)
Backup and recovery

• In general, backups should not be kept for a long time

 Because the data copies are only relevant in the event of a disaster, organizations will
typically have little use to restore a data backup that is more than a few weeks old
 Restoring a backup takes you back in time
 Like a time machine, but without the rest of the world – like your business partners and
customers – going back in time as well
Backup and recovery

• A common mistake is to mix up backup with archiving

 Backup is about protection against data loss
 Archiving deals with long term data storage, in order to comply with law and
regulations

• Backups should not be used to view the status of information from the
past
 It should be possible to retrieve these statuses from the system itself
 No data should ever be deleted in a typical production system
 Older data could be archived to a secondary system or database
Backup and recovery

• Backups need to be made at a regular basis

 Usually daily
 Sometimes more often – every hour, or even continuously in highly critical
environments

• 3-2-1 rule:
 Keep three copies of your data
 on two different media types
 with one copy stored at a separate location
Backup and recovery

• Backups must be available at a secondary site for restore

 Experience with real world disasters shows it is good practice to have a distance of at
least 5 km between the main site and the backup data

• Apart from application data, a copy must be available on the secondary

site of:
 Operating system installation disks
 Printed procedures on how to build up a new system using the backups
 License keys of the software (including the restore software)
Backup and recovery

• Test the restore procedure at least once a year to ensure restores work as
planned
 Include building up new hardware
 Have restore procedures tested by a third party, or at least by people that have not
performed a restore before
 In case of a real disaster we cannot assume that systems managers are able to restore
data again

• Restore tests should be performed each month to ensure backup media

still work as expected
 Restore some files
 Do the tapes really contain the expected data?
Backup schemes

• A backup scheme describes what data is backed-up, when, and how

• Backup schemes can become very complex in large environments with
many applications
• Four basic backup schemes
Backup schemes

• Full backup
 A complete copy of all data
 Full backups are only created at relatively large intervals (like a week or a month)
 Creating them takes much time, disk or tape space, and bandwidth
 Restoring a full backup takes the least amount of time

• Incremental backup
 Save only newly created or changed data since the last backup, regardless of whether it
is a previous incremental backup or a full backup
 Restoring an incremental backup can take a long time
 Especially when the last full backup is many incremental backups ago

• Differential backup
 Save only newly created or changed data since the last full backup
 Restoring a differential backup is quite efficient, as it implies restoring a full backup and
only the most recent differential backup
Backup schemes

• Incremental forever backups

 Make an initial full backup, after which only incremental backups are sent to the
backup system
 Metadata about the increments are stored in the backup system, allowing the backup
system to compile a point in time restore from the increments

• Continuous Data Protection (CDP)

 Guarantees that every change in the data is also simultaneously made in the backup
system
 The RPO (Recovery Point Objective) is set to zero, because each change immediately
triggers a backup process
 Expensive technology, and therefore only used in specific situations
Backup data retention time

• Backup data retention time is the amount of time in which a given set of
data will remain available for restore
• Defines how long backups are kept and at which interval
• In practice, a Grandfather-Father-Son (GFS) based schedule is often used:
 Each day a backup is made
 After a week, there are seven backups, of which the oldest backup is renamed to a
weekly backup
 After the second week, the same is done and the daily backups of the week before are
deleted
 Now there are eight backups: seven daily, two weekly
 Every four weeks, the weekly backup is renamed as a monthly backup and the weekly
backups are reused
 The daily backups are the son, the weekly backups are the father, and the monthly
backups are the grandfather
Archiving

• Archiving is mostly done for compliancy and regulation reasons

• Example:
 US regulations require all medical records to be retained for 30 years after a person's
death
 This means that X-rays taken when a child was born must be kept for as much as 130
years!

• Noncompliance to law and regulation can lead to serious business

disruption, fines, and even jail time
Archiving

• Archived data is read-only to protect it from being altered

 Very important for regulatory compliance and non-repudiation
 Some archiving systems store data in an encrypted form and use digital signatures to
prove data is not tampered with
 Some systems allow data to be written to it for archiving, but disallow changing or
deleting data
 CD / DVD/ Blu-ray
 WORM tapes
Archiving

• Data must be kept in such a way that it is guaranteed the data can be read
after a long time
 Digital format (like a Microsoft Word file or a JPG file)
 Physical format (like a DVD or a magnetic tape)
 Storage environment (temperature, humidity)

• Use open standards for storing archived data

 Open standards are well documented
 Reading data will always be feasible, using emulation software if needed
 Storing all documents in structured human-readable XML text files is one way to ensure
data can be read for many decades

• Transfer data that is to be kept for a long time to the latest storage media
standard every 10 years
Storage performance
Disk performance

• Disk performance is dependent on:

 Disk rotation speed
 Seek times
 Interface protocol

• Some common examples of rotation delay:

Disk RPM Average rotational delay (ms)
5,400 5.6
7,200 4.2
10,000 3
15,000 2
Disk performance

• Disks cannot spin much faster than 15,000 RPM

 At this speed the velocity at the edge of a 3.5” disk is 250 km/h!
 Increasing this velocity would physically destroy the disk

• Seek time is the time it takes for the head to get to the right track
 Average seek times:
 3 ms for high-end disks
 9 ms for low-end disks
IOPS

• Input/output Operations Per Second (IOPS) is a measure of how many

read and write operations a disk can complete in one second

• Writing is typically a bit slower than reading

Disk type I OPS

Mechanical
< 500
disks
• Typical IOPS: 400,000 read
SSD
150,000 write
1,000,000 read
NVMe
130,000 write
RAID penalty

• In RAID sets multiple disks are used to form one virtual disk (LUN)
• Writing data on multiple disks introduces some delay, known as the RAID
penalty
• Penalties for various RAID configurations are
 RAID 0: no penalty
 RAID 1: penalty of 2
 RAID 10: penalty of 2
 RAID 5: penalty of 4
 RAID 6: penalty of 6
Interface throughput

• Storage performance is I nterface Speed (Gbit/ s) Speed (GB/ s)

also dependent on how I DE (Parallel ATA) 0.133

0.32
fast the interface can SCSI
6 0.768
move data from the disks SATA
1.5
SAS 12
to the systems 7
NVMe
consuming the data and FC 128 16
vice versa
• An overview of the
various interface speeds:
Caching

• A caching system in disk controllers can improve performance by several

orders of magnitude
 Read-cache acts as a buffer for reads. When the same data is read multiple times, it is
served from cache
 Write-through cache: data is written to cache and then to disk, and only acknowledged
as written when the data is physically written on the disk
 Write-through cache: allows the disk controller to acknowledge the data as written as
soon as it is held in cache. This allows the cache to buffer writes quickly and then write
the data to the slower disk when the disk is ready to accept new I/O operations

• The type and amount of cache needed depends on what applications need
 A web server, for instance, will mostly benefit from read-cache, whereas most
databases are better off with write cache
Storage tiering

• Tiered storage creates a hierarchy of storage media, based on cost,

performance requirements, and availability requirements
• Example:
 Tier 1: Production data (SSD and SAS disks)
 Tier 2: Seldom used data, like email archives (NL-SAS disks)
 Tier 3: Backups (Virtual Tape Libraries on NL-SAS disks)
 Tier 4: Archived data (Tape or NL-SAS disks)

• The more tiers are used, the more effort it takes to manage the tiers
• Automated tiering usually checks for file access times, file creation date, and
file ownership, and automatically moves data to the storage medium that fits
best
• Storage tiering is especially important when storing data in the public cloud,
as it has different storage costs for each tier
Load optimization

• Storage performance is highly dependent on the type of load

• Most vendors recommend a specific storage configuration for their
systems or applications
 For example, Oracle recommends a combination of RAID 1 and 5 for its database in
order to optimize performance
Storage security
Protecting data at rest

• Data can be:

 In transit (transported over a network)
 In use (by an application or a cache)
 At rest (on a disk or a tape)

• Data at rest can be secured using encryption techniques

 Prevent reading or writing data to disk or tape without the correct
encryption/decryption key

• Disk encryption in the datacenter has limited benefits:

 Databases and applications need to work with unencrypted data to perform useful
work
 Disk encryption is only useful when the disks are physically lost or stolen (laptops,
desktops, or removable media)
 Disks in the datacentre are in a physically secure area
Protecting data at rest

• Disk encryption in the datacenter is useful:

 A disk drive might get in the wrong hands – for instance because it was removed after it
was marked "faulty" and was never destroyed
 In case of disk failure, having the data encrypted solves the issue of having potentially
sensitive data on a disk that can't be accessed anymore, as it is defective
 Maintenance contracts often require that a failed disk must be sent back to the vendor
after replacing it with a new one. Without disk encryption, returning disks may not be
possible since a failed disk cannot be erased anymore.
 Full disk encryption makes it harder for an attacker to retrieve data from the "empty"
space on the disks, which often contains traces of previously stored data.
Protecting data at rest

• Self-Encrypting Drives (SEDs):

 Use in laptops and desktops
 When an SED is powered up, authentication is required to access data – the user must
type in a password to start the boot sequence of the computer
 Encryption is built into the disk drive’s hardware
 Encryption keys are stored on the disk

• Cryptographic Disk Erasure (CDE):

 Deletes the encryption key on the disk
 This has the same effect as erasing all disk contents
 Without the key, unencrypted data can no longer be read from the disk
 One of the best ways to fully wipe a disk’s contents
SAN zoning

• SAN zoning is a method of arranging Fibre Channel devices into logical

groups on a SAN fabric for security purposes
 SAN zoning is implemented in the SAN switches
 SAN zones are comparable with VLANs in Ethernet networks
 Fibre Channel devices can only communicate with each other if they are members of
the same zone
SAN LUN masking

• In a SAN, LUN masking makes a LUN available to some hosts and

unavailable to other hosts
• LUN masking is implemented primarily at the HBA level, not in the SAN
switches
• It is good practice to use a combination of SAN zoning and LUN masking