IT-Symposium 2007 17.04.

2007

Storage Systems Division

Storage Virtualization with

IBM SAN Volume Controller

Torsten Rothenwaldt
IBM European Storage Center

DECUS IT-Symposium 2007 | 1G04 © 2007 IBM C orporation

Storage Systems Division

Agenda

Review different architectures of disk block virtualization.

Explain implementation details of the IBM SAN Volume Controller

as example of in-band appliance.

Discuss current enterprise usage of disk/file virtualization.

2 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 1
IT-Symposium 2007 17.04.2007

Storage Systems Division

Table of contents

Disk block virtualization

IBM SAN Volume Controller

Storage virtualization in enterprises today

3 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Disk (Drive) Virtualization

0000
0001
0002
0003
….
nnnn

Physical data layout
Virtual data layout

Cylinder/Head/Sector addresses Logical Block Addresses (LBA)

Media defects
“Defect-free”

4 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 2
IT-Symposium 2007 17.04.2007

Storage Systems Division

Block-Level Virtualization

Bound to a physical machine
Anywhere (can be moved or

replaced)

Fixed size and (pseudo-)
geometry
Size as needed (can grow,
shrink, or morph)

Limited performance

Performance scaling

Do break (occasionally)

Reliable as needed

Where does block-level virtualization reside?

5 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Virtualization in the Storage Subsystem

Provisioning
Per-host

Virtualization
RAID subsystem

+Heterogeneous hosts - Homogeneous storage

+Mature industry & products - Management
Performance Device-specific
Stable & reliable “per-box”
Perceived security

6 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 3
IT-Symposium 2007 17.04.2007

Storage Systems Division

Virtualization in the Host

Provisioning
Per-host…still

Virtualization
Host

RAID subsystem

+Flexibility (any storage) - Server-centric management

+File system coupling of RAID subsystems
(online growth, re-layout, of volumes
movement, snapshots,…) - Complexity
shared data clusters

7 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

“Out-of-Band” Virtualization
Provisioning
Data center-wide

Virtualization
Host
“SAN appliance”
separate box
Infrastructure In-host
RAID subsystem

+Data center-wide - May be host-invasive

management Host-specific
+Shorter I/O path - Appliance availability
+Light-weight compared to - Secure protocols
full volume manager

8 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 4
IT-Symposium 2007 17.04.2007

Storage Systems Division

Virtualization in the Network: “In-Band”

Provisioning
Data center-wide

SAN
Virtualization Host access
Device access
Host

Infrastructure

RAID subsystem

+Data center-wide - Complexity

management User (un)familiarity
Heterogeneous storage Integration needs
Heterogeneous hosts Needs clustering
- Performance perceptions

9 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Virtualization Devices for In-Band

Server-based device (appliance)
Switch-based device (fabric

application)
Virtualizes a variety of physical storage
using different HBAs. One pool. Network optimized
Implements complex storage solutions High port counts
inexpensively
Expensive
Adds another layer (managed
Elaborated functions not available
separately)
Fabric integration critical
Less interoperability issues (no fabric
integration) Standardization just beginning (ANSI
T11 Fabric Application Interface
Appears as standard device.
Standard)

10 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 5
IT-Symposium 2007 17.04.2007

Storage Systems Division

Table of contents

Disk block virtualization

IBM SAN Volume Controller

Storage virtualization in enterprises today

11 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Example of a Block-Level In-Band Appliance:

IBM SAN Volume Controller

J. S. Glider, C. F. Fuente, W. J. Scales:

The software architecture of a SAN storage control system.
IBM Systems Journal, Vol. 42 (2003) Nr. 2, pp. 232-249
( http://www.research.ibm.com/journal/ )

IBM Redbooks:
SG24-6423 IBM System Storage SAN Volume Controller
SG24-7275 Implementing the SVC in an OEM Environment
SG24-7371 Using the SVC for Business Continuity
( http://www.redbooks.ibm.com/ )

12 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 6
IT-Symposium 2007 17.04.2007

Storage Systems Division

IBM SVC: Architecture

Storage
Storage
Netw
Network
ork
Virtual LUNs Virt ual LUNs Virtu al LUNs Virt ual LUNs

N ode
Node Node N ode
Node Node Node N ode
Node Node
N ode Node

M anaged Disks

13 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

IBM SVC: Zoning

Host Zone:
-Hosts zoned only to SVC
-See only Virtual Disks that
SVC allows to see

Device Zone:
-Devices zoned only to SV
-See only SVC nodes as
connected hosts

14 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 7
IT-Symposium 2007 17.04.2007

Storage Systems Division

IBM SVC: Managed Disks

SVC does not perform RAID functions

Utilizes RAID capability of backend storage server

RAID-5, RAID-10, or RAID-1 recommended

Normally LUNs "surfaced" from storage systems are what the hosts on the
SAN see as physical disks

Disks surfaced by the storage systems are discovered by SVC and referred
to as Managed Disks (mdisks)

Spare capacity on mdisks can be reallocated transparently and
dynamically

R5 R 10 R5 R1 R5 R1
MD MD MD MD MD MD

SA NVC

R5 R5 R1 R5 R1
R 10 VLUN
VLUN VLUN VLUN VLUN VLUN

R AID C ontrolle r R AID C ontrolle r

15 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

IBM SVC: Managed Disk Groups

Once the mdisks are available to SVC, the user assigns them to one or more
pools called Managed Disk Groups

These MDGs are addressed by the SVC in terms of extents:
Extent size is determined at MDG creation time, default 16 MB, max 512 MB
Extent size determines maximum amount of storage SVC can manage
16-MB extents = 64 TB, 512-MB extents = 2 PB
Cannot migrate vdisks between MDGs with different extent sizes
Can migrate extents from mdisk to mdisk

R1 Pool

Ex te nt 1
R1
Ex te nt 2 R1
R1
Ex te nt 3
R5 Pool
Ex te nt 4
R10 Pool
Ex te nt 5
R5
R5 R5
... R10
R10
R10

Ex te nt n

16 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 8
IT-Symposium 2007 17.04.2007

Storage Systems Division

IBM SVC: Virtual Disks

From these extents, the user can build "virtual disks"

Various policies can be used to build them:
Striped — taking an extent in turn from each disk in the pool or a subset of the
disks in the pool (Virtual Disk striped across multiple disks)
Sequential — using a single disk in the pool (Virtual Disk mapped sequentially to
managed disk)
Image Mode — Virtual Disk = Physical LUN
SVC Cache — Enabled or Disabled

Real physical capacity must be available to create a vdisk

Virtual disks can be expanded, reduced, or deleted

IO governing can be enabled to limit IO/s or MB/s
Extent 1a
Extent 2a
Extent 1a Extent 2a Extent 3a
Extent 3a
Extent 1b Extent 2b Extent 3b Create a Extent 1b
Extent 1c Extent 2c Extent 3c
striped Extent 2b
Extent 1d Extent 2d Extent 3d
Extent 1e Extent 2e Extent 3e virtual disk Extent 3b
Extent 1c
Extent 1f Extent 2f Extent 3f
Extent 2c
Extent 1g Extent 2g Extent 3g
Extent 3c

A host vdisk is a
collection of Extents —
each 16 MB - 512 MB
17 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Cache Disabled Virtual Disks

Can be used with Image Mode VDisks only to enable use of copy services
of underlying disk subsystem

Can be used with striped mode VDisks
May be useful to preserve cache resources
Example: Use for TSM/HSM disk pools
Example: Use for test/dev VDisks in mixed production environment

Cache Disabled Virtual Disks are basically normal VDisks and can utilize
SVC capabilities like migration and copy services
Can use SVC copy services on the source VDisk
Cannot use SVC copy services on the target VDisk

Provide a way for parallel direct access to the underlying disk subsystem

18 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 9
IT-Symposium 2007 17.04.2007

Storage Systems Division

IBM SVC: IO Groups and Hosts

Each virtual disk is assigned to a particular IO Group (node pair).
Every node in the cluster is aware of the virtual disk, but only owning IO Group
services requests.
All IO is targeted at either of the nodes in the IO Group for purposes of caching
and load balancing.

It is these virtual disks that SVC presents to hosts on the SAN as targets
of IO.
The virtual disks are mapped to hosts (SDD for mutli-path operation).
Can be mapped to multiple hosts for use with clustering software.
The hosts see these as physical disks (in terms of the OS).
SVC knows hosts as groups of HBA WWPNs.

IO WWPN1

Group WWPN2

VDisk
Host
WWPN3

WWPN4

19 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

SVC Combined Physical & Logical View

Virtual Disks Mapped to Hosts

Virtual vdisk0 vdisk1 vdisk2 vdisk3 vdisk4

Disk 20GB 20GB 20GB 20GB 20GB

Virtual Disks are associated with

particular I/O Groups

SVC I/O Group 1 I/O Group 2

Cluster
Managed Disk Groups are accessible by
all I/O Groups in the Cluster.
Managed
Disk mdiskgrp0 [FA StT Group] - 40GB mdiskgrp1 [ESS Group] - 60GB
Groups
Managed mdisk0 mdisk1 mdisk2 mdisk3 mdisk4 mdisk5 mdisk6
Disk 10GB 10GB 10GB 10GB 20GB 20GB 20GB

FA StT FA StT FA StT FA StT ESS ESS ESS

LUN
10GB 10GB 10GB 10GB 20GB 20GB 20GB

20 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 10
IT-Symposium 2007 17.04.2007

Storage Systems Division

IBM SVC: Clustering

Cluster comprised of 2…8 nodes but administered as single image

No Linux clustering software, failover/failback is function of SVC code

One node automatically designated config/boss node for cluster

Assigned cluster IP address

Responsible for coordination of node transitions

Automatic failover of config node and IP address

Auto-restart of a node on failure and re-admission to cluster

Cluster requires majority of nodes remain operating to ensure quorum

Two nodes is a special case

Quorum disk is elected as a tie-breaker

Quorum disks are existing mdisks under SVC and use 1 extent

A node stores a write in its own write cache and the write cache of its partner node
before acknowledging to the host application

On node failure, surviving node empties write cache and proceeds in write-through
mode

21 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

IBM SVC: Copy Services

VD 1
VD 2

VD 6

VD 7

VD 1

VD 2
3

VD 6

VD 7
MD 1

VD 2

VD 3

VD 4

MD 5

MD 6

MD 7

MD 8

MD 1

VD 2

VD 3

VD 4

MD 5

MD 6

MD 7

MD 8

IO Group IO Group
MD

MD

MD

MD

MD

MD
LUN 1

LUN 2

LUN 3

LUN 4

LUN 1

LUN 2

LUN 3

LUN 4

LUN 1

LUN 2

LUN 3

LUN 4

LUN 1

LUN 2

LUN 3

LUN 4

RAID RAID RAID RAID

controller 1 controller 2 controller 3 controller 4

SANMigration
SAN Data Peer-to-Peer Remote
SANCopy
FlashCopy
“outside the(PPRC)
box” “outside the
“outside
box”the box”
(Metro / Global Mirror)

22 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 11
IT-Symposium 2007 17.04.2007

Storage Systems Division

IBM SVC Global Mirror

Long distance asynchronous remote mirroring function

Practically unlimited distances for business continuity

Does not wait for secondary I/O before completing host I/O
Minimizes performance impact to applications

Designed to maintain consistent secondary copy at all times
Once initial copy has completed

Built on Metro Mirror code base

Metro and Global Mirror
delivered as single feature
Offers great implementation
flexibility
No additional license charge
for existing MM users

23 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Global Mirror: Overview

Single remote copy partnership per cluster

Supports consistency groups that span I/O groups within cluster

Consistency group consists of MM or GM relationships not both

Cluster 1 Partnership Cluster 2

Vdisk Vdisk
Relationship
A C

Consist-
ency
Group
Vdisk Vdisk
Relationship
B D

24 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 12
IT-Symposium 2007 17.04.2007

Storage Systems Division

Global Mirror: Algorithm

I/O complete acknowledged to host at primary once following occurs:
Data is committed to cache in both nodes in I/O group
Log of I/O and its sequence number committed on both nodes of I/O group

Remote I/O is transmitted concurrently with processing of local I/O
Local host I/O normally completes before remote I/O completes

Extra messages passed between nodes in cluster

Used to detect concurrent host I/O among VDisks in same consistency group

Concurrent I/O is assigned a shared “sequence number”
During high loads many I/Os will share sequence number to minimize overhead

Remote I/Os are applied at secondary in “sequence number” order
Writes within a “sequence number” are non-dependant and written in any order once
received at secondary
Many sequence number “batches” can be outstanding on the long distance link
At secondary site a single sequence number is applied at a time

25 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Global Mirror: Link Tolerance

Existing redundancy used to continue operation if node or link fails:
Recovery begins by re-driving writes that were in flight.
Log of I/O established during initial host write used to support recovery.
Recovery uses read to primary VDisk to capture data for replay of write

Autonomic Link Tolerance feature to prevent performance problems leading

to application outage:
Monitors write latency impact of GM and considers stopping a GM relationship or
consistency group if impact exceeds a threshold
Link tolerance threshold allows ‘bursts’ of work to be tolerated
Defines period of time poor performance can be tolerated before GM stops
Settable from 1 minute to 1 day in 10 second increments (default 5 minutes)
Disabled by setting to ‘0’

26 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 13
IT-Symposium 2007 17.04.2007

Storage Systems Division

Table of contents

Disk block virtualization

IBM SAN Volume Controller

Storage virtualization in enterprises today

27 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Why To Use Storage Virtualization

The management nightmare

Too many servers, operating systems, storage systems, management
consoles, …
Too complex policies
Too large migration projects

Availability requirements

Storage resource utilization

Storage performance

28 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 14
IT-Symposium 2007 17.04.2007

Storage Systems Division

SVC Version 4.1 Supported Environments

HP/UX 11iV2 Linux
TRU64 (Intel/Power/zLinux)
Microsoft IBM Sun
Novell VMWare OpenVMS RHEL/SUSE
IBM
MSCS AIX Solaris 10 ServiceGuard W / LVM
NetWare Win / NW MPIO, VSS, GDS BladeCenter 1024
HACMP/XD VCS Clustering Clustering RHEL AS 4
Clustering guests x64 GPFS / VIO SUN Cluster Ia32, x64, System p Win/Linux/VMWare/AIX Hosts
OPM/FCS/IBS

Cisco
... McData

iSCSI to hosts
Via Cisco IPS SAN with 4Gbps fabric SAN
Continuous Copy
Point-in-time Copy
Full volume Metro Mirror
Copy on write Global Mirror

SAN SAN
Volume Controller Volume Controller

IBM IBM IBM Hitachi Hitachi Hitachi HP HP HP EMC EMC/Dell Sun NetApp STK
ESS, DS N series Lightning Thunder Tagma EVA MA/EMA XP Symm CLARiiON FAS D173, 178, 220,
FAStT DS4K / 6K 5200, 5500 9980V 9200 Store 3000/5000 8000 48, 128 8000 FC4700 9910/9960 3020, 3050 240, 280
DS8000 9970V 95xxV 4000/6000 12000 512, 1024 DMX CX2/3/4/5/6/700 9970/9980 FLX210, 240,
USP, NSC55
DS4800 9910/9960 9520V 8000 16000 280, 380

29 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Performance: SVC SPC-1 Benchmark

http://www.storageperformance.org/results/a00043-r1
_IBM_SPC1_executive-summary.pdf

30 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 15
IT-Symposium 2007 17.04.2007

Storage Systems Division

Performance: SPC-2 Benchmark

Newest addition to the Storage Performance Council Benchmarks

Composite of three workloads to measure sequential performance
Large file processing – scientific and large-scale financial processing
Large database queries – data mining and business intelligence
Video on demand – streaming movies to end users

SVC posted the highest results in this industry-recognized storage
performance test
SVC (3.517 GB/s) + DS8300 (3.217 GB/s) lead the industry in this new benchmark

31 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Enterprise Usage Today

Widely accepted by customers:

Tape drive and library virtualization
Disk block level virtualization in LVM and storage subsystem
In-band file system virtualization (NAS)
Aggregated networks (iSCSI, long distance connections)

Beginning of the lifecycle:

Disk block fabric-layer virtualization (in-band and out-of-band)

Niche markets (this may change):

Virtual SANs
File and record virtualization

32 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 16
IT-Symposium 2007 17.04.2007

Storage Systems Division

Why Do Customers Hesitate?

Reliability
SAN and storage failures may affect the whole datacenter.
SAN and storage failures may destroy data.
Security implications not clear.

Complexity
Lack of standards.
Products difficult to understand and evaluate.
Multi-vendor support.

Market situation
Religious wars about architectures.
Too many different solutions.

33 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

Storage Systems Division

Disclaimer Information
Copyright © 2007 by Inte rnatio na l B us iness M achines Corporation.

No part o f this document ma y be reprod uced or trans mitted in any form w ithout writte n permissio n from IBM Corporation.

All stateme nts regarding IBM 's future dire ctio n and intent are subje ct to cha nge or withdra wa l without notice, and
represent goals and objective s only.

Refere nce s in this document to curre nt or future IBM produc ts, progra ms, or services does not imp ly tha t IBM intend s to make
suc h s uch products, programs or services availab le in a ll countries in whic h IBM operates or does bus iness.

THE INFORMATION PRO VIDE D IN THIS DO CUM EN T IS DISTRIBUT ED "AS IS" WITHOUT ANY WAR RAN TY,
EIT HER EXPR ESS O R IMPLIED. IBM EXP RESSLY DISCLAIMS AN Y WAR RANTIES O F MERC HAN TABILIT Y,
FITNESS FO R A PARTICU LAR PURPOS E OR IN FRIN GEM EN T. IBM shall ha ve no responsib ility to upda te this
info rmatio n. IBM products are warra nted accord ing to the terms and cond itions of the agree me nts (e.g., IBM Custo mer
Agree me nt, Statement of Limited Warra nty, Interna tional Progra m License Agreement, etc.) unde r whic h they are provided.

T he provisio n o f the informatio n contained herein is no t intended to, and does not, gra nt any right or lice nse under any IBM
pate nts or copyrights. Inquiries regarding patent or copyright licenses sho uld be made, in writing, to :

IBM D irector of L ice nsing

IBM C orporatio n
North Castle Drive
Armonk, NY 10504-1785
U.S.A.

34 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 17
IT-Symposium 2007 17.04.2007

Storage Systems Division

Closing slide

Thank you

DECUS IT-Symposium 2007 | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 18