Red Hat Enterprise Linux 7

Storage Administration Guide

Deploying and configuring single-node storage in Red Hat Enterprise Linux

7

Milan Navrtil Jacquelynn East Don Domingo

Red Hat Subject Matter ExpertsJosef Bacik
Kamil Dudka Hans de Goede Harald Hoyer
Doug Ledford Daniel Novotny Nathan Straz
David Wysochanski Contributors Michael Christie
Sachin Prabhu Rob Evers David Howells
David Lehman Jeff Moyer Eric Sandeen
Mike Snitzer
Red Hat Enterprise Linux 7 Storage Administration Guide

Deploying and configuring single-node storage in Red Hat Enterprise Linux

7

Milan Navrtil
Red Hat Custo mer Co ntent Services

Jacquelynn East
Red Hat Custo mer Co ntent Services

Do n Do mingo
Red Hat Custo mer Co ntent Services

Jo sef Bacik
Server Develo pment Kernel File System
[email protected] m
Disk Quo tas

Kamil Dudka
Base Operating System Co re Services - BRNO
[email protected] m
Access Co ntro l Lists

Hans de Go ede
Base Operating System Installer
hdego [email protected] m
Partitio ns

Harald Ho yer
Engineering So ftware Engineering
[email protected] m
File Systems

Do ug Ledfo rd
Server Develo pment Hardware Enablement
dledfo [email protected] m
RAID

Daniel No vo tny
Base Operating System Co re Services - BRNO
dno vo [email protected] m
The /pro c File System

Nathan Straz
Quality Engineering QE - Platfo rm
[email protected] m
GFS2

David Wyso chanski

Server Develo pment Kernel Sto rage
dwyso [email protected] m
LVM/LVM2
Michael Christie
Server Develo pment Kernel Sto rage
[email protected] m
Online Sto rage

Sachin Prabhu
So ftware Maintenance Engineering
[email protected] m
NFS

Ro b Evers
Server Develo pment Kernel Sto rage
[email protected] m
Online Sto rage

David Ho wells
Server Develo pment Hardware Enablement
dho [email protected] m
FS-Cache

David Lehman
Base Operating System Installer
[email protected] m
Sto rage co nfiguratio n during installatio n

Jeff Mo yer
Server Develo pment Kernel File System
jmo [email protected] m
So lid-State Disks

Eric Sandeen
Server Develo pment Kernel File System
[email protected] m
ext3, ext4 , XFS, Encrypted File Systems

Mike Snitzer
Server Develo pment Kernel Sto rage
[email protected] m
I/O Stack and Limits

Red Hat Subject Matter Experts

Co ntributo rs

Edited by
Marek Suchnek
Red Hat Custo mer Co ntent Services
[email protected] m
Legal Notice
Co pyright 20 17 Red Hat, Inc.

This do cument is licensed by Red Hat under the Creative Co mmo ns Attributio n-ShareAlike 3.0
Unpo rted License. If yo u distribute this do cument, o r a mo dified versio n o f it, yo u must pro vide
attributio n to Red Hat, Inc. and pro vide a link to the o riginal. If the do cument is mo dified, all Red
Hat trademarks must be remo ved.

Red Hat, as the licenso r o f this do cument, waives the right to enfo rce, and agrees no t to assert,
Sectio n 4 d o f CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shado wman lo go , JBo ss, OpenShift, Fedo ra, the Infinity
lo go , and RHCE are trademarks o f Red Hat, Inc., registered in the United States and o ther
co untries.

Linux is the registered trademark o f Linus To rvalds in the United States and o ther co untries.

Java is a registered trademark o f Oracle and/o r its affiliates.

XFS is a trademark o f Silico n Graphics Internatio nal Co rp. o r its subsidiaries in the United
States and/o r o ther co untries.

MySQL is a registered trademark o f MySQL AB in the United States, the Euro pean Unio n and
o ther co untries.

No de.js is an o fficial trademark o f Jo yent. Red Hat So ftware Co llectio ns is no t fo rmally

related to o r endo rsed by the o fficial Jo yent No de.js o pen so urce o r co mmercial pro ject.

The OpenStack Wo rd Mark and OpenStack lo go are either registered trademarks/service

marks o r trademarks/service marks o f the OpenStack Fo undatio n, in the United States and o ther
co untries and are used with the OpenStack Fo undatio n's permissio n. We are no t affiliated with,
endo rsed o r spo nso red by the OpenStack Fo undatio n, o r the OpenStack co mmunity.

All o ther trademarks are the pro perty o f their respective o wners.

Abstract
This guide pro vides instructio ns o n ho w to effectively manage sto rage devices and file systems
o n Red Hat Enterprise Linux 7. It is intended fo r use by system administrato rs with basic to
intermediate kno wledge o f Red Hat Enterprise Linux o r Fedo ra.
 T able of Cont ent s

T able of Contents
C
. .hapt
. . . .er
. .1. .. O
. .verview
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5. . . . . . . . . .
1.1. What' s New in Red Hat Enterp ris e Linux 7 5

P
. .art
. . .I.. File
. . . .Syst
. . . .ems
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7. . . . . . . . . .

C
. .hapt. . . .er
. .2. .. File
. . . .Syst
. . . .em
. . .St
. .ruct
. . . ure
. . . .and
. . . Maint
. . . . . enance
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8. . . . . . . . . .
2 .1. O verview o f Files ys tem Hierarc hy Stand ard (FHS) 8
2 .2. Sp ec ial Red Hat Enterp ris e Linux File Lo c atio ns 15
2 .3. The /p ro c Virtual File Sys tem 16
2 .4. Dis c ard unus ed b lo c ks 16

C
. .hapt
. . . .er
. .3.
. .T. he
. . .XFS
. . . .File
. . . Syst
. . . . em
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 7. . . . . . . . . .
3 .1. Creating an XFS File Sys tem 18
3 .2. Mo unting an XFS File Sys tem 19
3 .3. XFS Q uo ta Manag ement 19
3 .4. Inc reas ing the Siz e o f an XFS File Sys tem 22
3 .5. Rep airing an XFS File Sys tem 22
3 .6 . Sus p end ing an XFS File Sys tem 23
3 .7. Bac kup and Res to ratio n o f XFS File Sys tems 23
3 .8 . O ther XFS File Sys tem Utilities 25
3 .9 . Mig rating fro m ext4 to XFS 26

C
. .hapt
. . . .er
. .4. .. T. he
. . . Ext
. . . 3. .File
. . . Syst
. . . . em
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
...........
4 .1. Creating an Ext3 File Sys tem 31
4 .2. Co nverting to an Ext3 File Sys tem 31
4 .3. Reverting to an Ext2 File Sys tem 32

C
. .hapt . . . .er
. .5.
. .T. he
. . .Ext
. . .4. .File
. . . Syst
. . . . em
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
...........
5 .1. Creating an Ext4 File Sys tem 34
5 .2. Mo unting an Ext4 File Sys tem 35
5 .3. Res iz ing an Ext4 File Sys tem 36
5 .4. Bac kup ext2/3/4 File Sys tems 37
5 .5. Res to re an ext2/3/4 File Sys tem 38
5 .6 . O ther Ext4 File Sys tem Utilities 40

C
. .hapt
. . . .er
. .6. .. Bt
. . rfs
. . . (T
. .echnology
. . . . . . . . . .Preview)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4. 1. . . . . . . . . .
6 .1. Creating a b trfs File Sys tem 41
6 .2. Mo unting a b trfs file s ys tem 41
6 .3. Res iz ing a b trfs file s ys tem 42
6 .4. Integ rated Vo lume Manag ement o f Multip le Devic es 45
6 .5. SSD O p timiz atio n 49
6 .6 . b trfs referenc es 49

C
. .hapt
. . . .er
. .7. .. G
. .lobal
. . . . File
. . . .Syst
. . . .em
. . .2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
...........

C
. .hapt . . . .er
. .8. .. Net
. . . work
. . . . .File
. . . Syst
. . . . em
. . . (NFS)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
...........
8 .1. Ho w NFS Wo rks 52
8 .2. p NFS 55
8 .3. NFS Client Co nfig uratio n 55
8 .4. auto fs 57
8 .5. Co mmo n NFS Mo unt O p tio ns 63
8 .6 . Starting and Sto p p ing NFS 64
8 .7. NFS Server Co nfig uratio n 65
8 .8 . Sec uring NFS 72

1
St orage Administ rat ion G uide

8 .9 . NFS and rp c b ind 74

8 .10 . Referenc es 75

C
. .hapt
. . . .er
. .9. .. FS-
. . . Cache
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7. 7. . . . . . . . . .
9 .1. Perfo rmanc e G uarantee 78
9 .2. Setting Up a Cac he 78
9 .3. Us ing the Cac he With NFS 79
9 .4. Setting Cac he Cull Limits 81
9 .5. Statis tic al Info rmatio n 82
9 .6 . Referenc es 82

P
. .art
. . .II.. .St
. .orage
. . . . . Administ
. . . . . . . .rat
. . ion
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8. 3. . . . . . . . . .

C
. .hapt
. . . .er
. .1. 0. .. St
. . orage
. . . . . .Considerat
. . . . . . . . . ions
. . . . .During
. . . . . . Inst
. . . .allat
. . . .ion
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8. 4. . . . . . . . . .
10 .1. Sp ec ial Co ns id eratio ns 84

C
. .hapt
. . . .er
. .1. 1. .. File
. . . .Syst
. . . .em
. . .Check
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8. 6. . . . . . . . . .
11.1. Bes t Prac tic es fo r fs c k 86
11.2. Files ys tem-Sp ec ific Info rmatio n fo r fs c k 87

C
. .hapt
. . . .er
. .1. 2. .. Part
. . . .it. ions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9. 1. . . . . . . . . .
M anip ulating Partitio ns o n Devic es in Us e 91
M o d ifying the Partitio n Tab le 91
12.1. Viewing the Partitio n Tab le 92
12.2. Creating a Partitio n 93
12.3. Remo ving a Partitio n 95
12.4. Setting a Partitio n Typ e 96
12.5. Res iz ing a Partitio n with fd is k 96

C
. .hapt
. . . .er
. .1. 3.
. . Creat
. . . . .ing
. . . and
. . . .Maint
. . . . .aining
. . . . . .Snapshot
. . . . . . . . .s. wit
. . .h. Snapper
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9. 9. . . . . . . . . .
13.1. Initial Snap p er Setup 99
13.2. Creating a Snap p er Snap s ho t 10 0
13.3. Trac king Chang es Between Snap p er Snap s ho ts 10 3
13.4. Revers ing Chang es in Between Snap s ho ts 10 6
13.5. Deleting a Snap p er Snap s ho t 10 8

C
. .hapt
. . . .er
. .1. 4. .. Swap
. . . . . Space
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.0. 9. . . . . . . . . .
14.1. Ad d ing Swap Sp ac e 110
14.2. Remo ving Swap Sp ac e 111
14.3. Mo ving Swap Sp ac e 113

C
. .hapt
. . . .er
. .1. 5.
. . Syst
. . . . em
. . . St
. . orage
. . . . . .Manager
. . . . . . . .(SSM)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.1. 4. . . . . . . . . .
15.1. SSM Bac kend s 114
15.2. Co mmo n SSM Tas ks 116
15.3. SSM Res o urc es 123

C
. .hapt
. . . .er
. .1. 6. .. Disk
....Q. .uot
. . . as
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.2. 4. . . . . . . . . .
16 .1. Co nfig uring Dis k Q uo tas 124
16 .2. Manag ing Dis k Q uo tas 128
16 .3. Dis k Q uo ta Referenc es 130

C
. .hapt
. . . .er
. .1. 7. .. Redundant
. . . . . . . . . . .Array
. . . . .of
. . Independent
. . . . . . . . . . . .Disks
. . . . . (RAID)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 32
...........
17.1. RAID Typ es 132
17.2. RAID Levels and Linear Sup p o rt 133
17.3. Linux RAID Sub s ys tems 135
17.4. RAID Sup p o rt in the Ins taller 136
17.5. Co nverting Ro o t Dis k to RAID1 after Ins tallatio n 136

2
 T able of Cont ent s

17.5. Co nverting Ro o t Dis k to RAID1 after Ins tallatio n 136

17.6 . Co nfig uring RAID Sets 136
17.7. Ad vanc ed RAID Devic e Creatio n 137

C
. .hapt
. . . .er
. .1. 8. .. Using
. . . . . .t.he
. . mount
. . . . . . .Command
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 39
...........
18 .1. Lis ting Currently Mo unted File Sys tems 139
18 .2. Mo unting a File Sys tem 140
18 .3. Unmo unting a File Sys tem 149
18 .4. mo unt Co mmand Referenc es 149

C
. .hapt
. . . .er
. .1. 9. .. T
. .he
. . volume_key
. . . . . . . . . . .Funct
. . . . . ion
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 51
...........
19 .1. vo lume_key Co mmand s 151
19 .2. Us ing vo lume_key as an ind ivid ual us er 152
19 .3. Us ing vo lume_key in a larg er o rg aniz atio n 153
19 .4. vo lume_key Referenc es 155

C
. .hapt
. . . .er
. .2. 0. .. Solid-
. . . . . .St
. .at. e
. .Disk
. . . . Deployment
. . . . . . . . . . .G
. .uidelines
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 56
...........
D ep lo yment Co ns id eratio ns 156
P erfo rmanc e Tuning Co ns id eratio ns 158

C
. .hapt
. . . .er
. .2. 1. .. Writ
. . . .e. Barriers
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 59
...........
2 1.1. Imp o rtanc e o f Write Barriers 159
2 1.2. Enab ling /Dis ab ling Write Barriers 159
2 1.3. Write Barrier Co ns id eratio ns 16 0

C
. .hapt
. . . .er
. .2. 2. .. St
. . orage
. . . . . .I/O
. . .Alignment
. . . . . . . . . and
. . . . Siz
. . .e. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.6. 2. . . . . . . . . .
2 2.1. Parameters fo r Sto rag e Ac c es s 16 2
2 2.2. Us ers p ac e Ac c es s 16 3
2 2.3. Stand ard s 16 4
2 2.4. Stac king I/O Parameters 16 5
2 2.5. Lo g ic al Vo lume Manag er 16 5
2 2.6 . Partitio n and File Sys tem To o ls 16 5

C
. .hapt
. . . .er
. .2. 3.
. . Set
. . . t.ing
. . . Up
. . .A
. .Remot
. . . . . .e. Diskless
. . . . . . . .Syst
. . . .em
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.6. 7. . . . . . . . . .
2 3.1. Co nfig uring a tftp Servic e fo r Dis kles s Clients 16 7
2 3.2. Co nfig uring DHCP fo r Dis kles s Clients 16 8
2 3.3. Co nfig uring an Exp o rted File Sys tem fo r Dis kles s Clients 16 9

C
. .hapt
. . . .er
. .2. 4. .. .O. nline
. . . . . St
. . orage
. . . . . .Management
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.7. 1. . . . . . . . . .
2 4.1. Targ et Setup 171
2 4.2. Creating an iSCSI Initiato r 179
2 4.3. Fib re Channel 18 0
2 4.4. Co nfig uring a Fib re Channel o ver Ethernet Interfac e 18 2
2 4.5. Co nfig uring an FCo E Interfac e to Auto matic ally Mo unt at Bo o t 18 4
2 4.6 . iSCSI 18 5
2 4.7. Pers is tent Naming 18 6
2 4.8 . Remo ving a Sto rag e Devic e 19 0
2 4.9 . Remo ving a Path to a Sto rag e Devic e 19 1
2 4.10 . Ad d ing a Sto rag e Devic e o r Path 19 2
2 4.11. Sc anning Sto rag e Interc o nnec ts 19 4
2 4.12. iSCSI Dis c o very Co nfig uratio n 19 5
2 4.13. Co nfig uring iSCSI O fflo ad and Interfac e Bind ing 19 6
2 4.14. Sc anning iSCSI Interc o nnec ts 20 0
2 4.15. Lo g g ing in to an iSCSI Targ et 20 2
2 4.16 . Res iz ing an O nline Lo g ic al Unit 20 3
2 4.17. Ad d ing /Remo ving a Lo g ic al Unit Thro ug h res c an-s c s i-b us .s h 20 6

3
St orage Administ rat ion G uide

2 4.17. Ad d ing /Remo ving a Lo g ic al Unit Thro ug h res c an-s c s i-b us .s h 20 6

2 4.18 . Mo d ifying Link Lo s s Behavio r 20 7
2 4.19 . Co ntro lling the SCSI Co mmand Timer and Devic e Status 210
2 4.20 . O nline Sto rag e Co nfig uratio n Tro ub les ho o ting 211
2 4.21. Co nfig uring Maximum Time fo r Erro r Rec o very with eh_d ead line 212

C
. .hapt
. . . .er
. .2. 5.
. . Device
. . . . . . Mapper
. . . . . . . Mult
. . . . ipat
. . . .hing
. . . . and
. . . . Virt
. . . ual
. . . .St
. .orage
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2. 1. 3. . . . . . . . . .
2 5.1. Virtual Sto rag e 213
2 5.2. DM-Multip ath 213

C
. .hapt
. . . .er
. .2. 6. .. Ext
. . . ernal
. . . . . Array
. . . . . Management
. . . . . . . . . . . .(libSt
. . . . .orageMgmt
. . . . . . . . . .) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2. 1. 5. . . . . . . . . .
2 6 .1. What is lib Sto rag eMg mt 215
2 6 .2. Termino lo g y fro m lib Sto rag eMg mt 216
2 6 .3. Ins tallatio n 217
2 6 .4. Us e o f the lib Sto rag eMg mt 218

A
. .ppendix
. . . . . . . A.
. . Red
. . . . Hat
. . . . Cust
. . . . omer
. . . . . Port
. . . .al
. .Labs
. . . . .Relevant
. . . . . . . .t.o. St
. . orage
. . . . . .Administ
. . . . . . . .rat
. . ion
. . . . . . . . . . . . . . . .2. 2. 3. . . . . . . . . .
S CSI d ec o d er 223
File Sys tem Layo ut Calc ulato r 223
L VM RAID Calc ulato r 223
i SCSI Help er 223
S amb a Co nfig uratio n Help er 223
M ultip ath Help er 224
N FS Help er 224
M ultip ath Co nfig uratio n Vis ualiz er 224
R HEL Bac kup and Res to re As s is tant 224

A
. .ppendix
. . . . . . . B.
. . .Revision
. . . . . . . .Hist
. . . ory
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2.2. 6. . . . . . . . . .

I.ndex
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2.2. 6. . . . . . . . . .

4
 Chapt er 1 . O verview

Chapter 1. Overview
The Storage Administration Guide contains extensive information on supported file systems and data
storage features in Red Hat Enterprise Linux 7. This book is intended as a quick reference for
administrators managing single-node (that is, non-clustered) storage solutions.

The Storage Administration Guide is split into two parts: File Systems, and Storage Administration.

The File Systems part details the various file systems Red Hat Enterprise Linux 7 supports. It
describes them and explains how best to utilize them.

The Storage Administration part details the various tools and storage administration tasks Red Hat
Enterprise Linux 7 supports. It describes them and explains how best to utilize them.

1.1. What 's New in Red Hat Ent erprise Linux 7

Red Hat Enterprise Linux 7 features the following file system enhancements:

eCrypt fs not included

As of Red Hat Enterprise Linux 7 eCryptfs is not included. Refer to Red Hat's Security Guide for
information on encrypting file systems.

Syst em St orage Manager

Red Hat Enterprise Linux 7 includes a new application called System Storage Manager. This
provides a command-line interface to manage various storage technologies. For more information,
see Chapter 15, System Storage Manager (SSM).

XFS is t he default File Syst em

As of Red Hat Enterprise Linux 7, XFS is the default file system. For more information about the XFS
file system, see Chapter 3, The XFS File System.

File syst em rest ruct ure

Red Hat Enterprise Linux 7 introduces a new file system structure. The directories /bi n, /sbi n,
/l i b, and /l i b6 4 are now nested under /usr.

Snapper

Red Hat Enterprise Linux 7 introduces a new tool called snapper that allows for the easy creation and
management of snapshots for LVM and BTRFS. For more information refer to Chapter 13, Creating and
Maintaining Snapshots with Snapper.

BT RFS (T echnology Preview)

BTRFS is a local file system that aims to provide better performance and scalability, including
integrated LVM operations. This file system is not fully supported by Red Hat and as such is a
technology preview. For more information on Btrfs, see Chapter 6, Btrfs (Technology Preview).

NFSv2 no longer support ed

5
St orage Administ rat ion G uide

As of Red Hat Enterprise Linux 7, NFSv2 is no longer supported.

6
 P art I. File Syst ems

Part I. File Systems

The File Systems section provides information on the file system structure and maintenance, the Btrfs
Technology Preview, and file systems that Red Hat fully supports: ext3, ext4, GFS2, XFS, NFS, and
FS-Cache.

For an overview of Red Hat Enterprise Linux file systems and storage limits, see Red Hat Enterprise
Linux technology capabilities and limits at Red Hat Knowledgebase.

XFS is the default file system in Red Hat Enterprise Linux 7 and Red Hat, and Red Hat recommends
you to use XFS unless you have a strong reason to use another file system. For general information
on common file systems and their properties, see the following Red Hat Knowledgebase article: How
to Choose your Red Hat Enterprise Linux File System.

7
St orage Administ rat ion G uide

Chapter 2. File System Structure and Maintenance

The file system structure is the most basic level of organization in an operating system. The way an
operating system interacts with its users, applications, and security model nearly always depends on
how the operating system organizes files on storage devices. Providing a common file system
structure ensures users and programs can access and write files.

File systems break files down into two logical categories:

Shareable vs. unsharable files

Variable vs. static files

Shareable files can be accessed locally and by remote hosts; unsharable files are only available
locally. Variable files, such as documents, can be changed at any time; static files, such as binaries,
do not change without an action from the system administrator.

Categorizing files in this manner helps correlate the function of each file with the permissions
assigned to the directories which hold them. How the operating system and its users interact with a
file determines the directory in which it is placed, whether that directory is mounted with read-only or
read/write permissions, and the level of access each user has to that file. The top level of this
organization is crucial; access to the underlying directories can be restricted, otherwise security
problems could arise if, from the top level down, access rules do not adhere to a rigid structure.

2.1. Overview of Filesyst em Hierarchy St andard (FHS)

Red Hat Enterprise Linux uses the Filesystem Hierarchy Standard (FHS) file system structure, which
defines the names, locations, and permissions for many file types and directories.

The FHS document is the authoritative reference to any FHS-compliant file system, but the standard
leaves many areas undefined or extensible. This section is an overview of the standard and a
description of the parts of the file system not covered by the standard.

The two most important elements of FHS compliance are:

Compatibility with other FHS-compliant systems

The ability to mount a /usr/ partition as read-only. This is especially crucial, since /usr/
contains common executables and should not be changed by users. In addition, since /usr/ is
mounted as read-only, it should be mountable from the CD -ROM drive or from another machine
via a read-only NFS mount.

2.1.1. FHS Organiz at ion

The directories and files noted here are a small subset of those specified by the FHS document. Refer
to the latest FHS documentation for the most complete information at http://www.pathname.com/fhs/.

Note

What directories are available depends on what is installed on any given system. The
following lists are only an example of what may be found.

2 .1 .1 .1 . Gat he ring File Syst e m Info rm at io n

8
 Chapt er 2 . File Syst em St ruct ure and Maint enance

The d f command reports the system's disk space usage. Its output looks similar to the following:

Examp le 2.1. d f co mman d o u t p u t

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/mapper/VolGroup00-LogVol00
11675568 6272120 4810348 57% / /dev/sda1
100691 9281 86211 10% /boot
none 322856 0 322856 0% /dev/shm

By default, d f shows the partition size in 1 kilobyte blocks and the amount of used and available
disk space in kilobytes. To view the information in megabytes and gigabytes, use the command d f -
h. The -h argument stands for " human-readable" format. The output for d f -h looks similar to the
following:

Examp le 2.2. d f -h co mman d o u t p u t

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00
12G 6.0G 4.6G 57% / /dev/sda1
99M 9.1M 85M 10% /boot
none 316M 0 316M 0% /dev/shm

Note

In the above examples, the mounted partition /d ev/shm represents the system's virtual
memory file system.

The d u command displays the estimated amount of space being used by files in a directory,
displaying the disk usage of each subdirectory. The last line in the output of d u shows the total disk
usage of the directory; to see only the total disk usage of a directory in human-readable format, use
d u -hs. For more options, refer to man d u.

To view the system's partitions and disk space usage in a graphical format, use the Gnome System
Mo ni to r by clicking on Ap p licat io n s Syst em T o o ls Syst em Mo n it o r or using the
command g no me-system-mo ni to r. Select the Fi l e Systems tab to view the system's partitions.
The figure below illustrates the Fi l e Systems tab.

9
St orage Administ rat ion G uide

Fig u re 2.1. G N O ME Syst em Mo n it o r File Syst ems t ab

2 .1 .1 .2 . T he /bo o t/ Dire ct o ry

The /bo o t/ directory contains static files required to boot the system, for example, the Linux kernel.
These files are essential for the system to boot properly.

Warning

D o not remove the /bo o t/ directory. D oing so renders the system unbootable.

2 .1 .1 .3. T he /d ev/ Dire ct o ry

The /d ev/ directory contains device nodes that represent the following device types:

devices attached to the system;

virtual devices provided by the kernel.

These device nodes are essential for the system to function properly. The ud evd daemon creates
and removes device nodes in /d ev/ as needed.

D evices in the /d ev/ directory and subdirectories are defined as either character (providing only a
serial stream of input and output, for example, mouse or keyboard) or block (accessible randomly,
such as a hard drive or a floppy drive). If GNOME or KD E is installed, some storage devices are
automatically detected when connected (such as with USB) or inserted (such as a CD or D VD drive),

10
 Chapt er 2 . File Syst em St ruct ure and Maint enance

and a pop-up window displaying the contents appears.

T ab le 2.1. Examp les o f co mmo n f iles in t h e /d ev d irect o ry

File D escrip t io n
/dev/hda The master device on the primary ID E channel.
/dev/hdb The slave device on the primary ID E channel.
/dev/tty0 The first virtual console.
/dev/tty1 The second virtual console.
/dev/sda The first device on the primary SCSI or SATA
channel.
/dev/lp0 The first parallel port.

A valid block device can be one of two types of entries:

A map p ed d evice

A logical volume in a volume group, for example, /d ev/mapper/Vo l G ro up0 0 -

Lo g Vo l 0 2.

A st at ic d evice

A traditional storage volume, for example, /d ev/sdbX, where sdb is a storage device name
and X is the partition number. /d ev/sd bX can also be /d ev/d i sk/by-i d /WWID, or
/d ev/d i sk/by-uui d /UUID, (see Section 24.7, Persistent Naming for more information
on both these options).

2 .1 .1 .4 . T he /etc/ Dire ct o ry

The /etc/ directory is reserved for configuration files that are local to the machine. It should contain
no binaries; any binaries should be moved to /usr/bi n/ or /usr/sbi n/.

For example, the /etc/skel / directory stores " skeleton" user files, which are used to populate a
home directory when a user is first created. Applications also store their configuration files in this
directory and may reference them when executed. The /etc/expo rts file controls which file systems
export to remote hosts.

2 .1 .1 .5 . T he /mnt/ Dire ct o ry

The /mnt/ directory is reserved for temporarily mounted file systems, such as NFS file system
mounts. For all removable storage media, use the /med i a/ directory. Automatically detected
removable media will be mounted in the /med i a directory.

Important

The /mnt directory must not be used by installation programs.

2 .1 .1 .6 . T he /o pt/ Dire ct o ry

The /o pt/ directory is normally reserved for software and add-on packages that are not part of the
default installation. A package that installs to /o pt/ creates a directory bearing its name, for
example, /o pt/packagename/. In most cases, such packages follow a predictable subdirectory
structure; most store their binaries in /o pt/packagename/bi n/ and their man pages in

11
St orage Administ rat ion G uide

/o pt/packagename/man/.

2 .1 .1 .7 . T he /pro c/ Dire ct o ry

The /pro c/ directory contains special files that either extract information from the kernel or send
information to it. Examples of such information include system memory, CPU information, and
hardware configuration. For more information about /pro c/, refer to Section 2.3, The /proc Virtual
File System .

2 .1 .1 .8 . T he /srv/ Dire ct o ry

The /srv/ directory contains site-specific data served by a Red Hat Enterprise Linux system. This
directory gives users the location of data files for a particular service, such as FTP, WWW, or CVS.
D ata that only pertains to a specific user should go in the /ho me/ directory.

2 .1 .1 .9 . T he /sys/ Dire ct o ry

The /sys/ directory utilizes the new sysfs virtual file system specific to the kernel. With the
increased support for hot plug hardware devices in the kernel, the /sys/ directory contains
information similar to that held by /pro c/, but displays a hierarchical view of device information
specific to hot plug devices.

2 .1 .1 .1 0 . T he /usr/ Dire ct o ry

The /usr/ directory is for files that can be shared across multiple machines. The /usr/ directory is
often on its own partition and is mounted read-only. At a minimum, /usr/ should contain the
following subdirectories:

/usr/bi n

This directory is used for binaries.

/usr/etc

This directory is used for system-wide configuration files.

/usr/g ames

This directory stores games.

/usr/i ncl ud e

This directory is used for C header files.

/usr/kerbero s

This directory is used for Kerberos-related binaries and files.

/usr/l i b

This directory is used for object files and libraries that are not designed to be directly
utilized by shell scripts or users.

As of Red Hat Enterprise Linux 7.0, the /l i b/ directory has been merged with /usr/l i b. It
now also contains libraries needed to execute the binaries in /usr/bi n/ and
/usr/sbi n/. These shared library images are used to boot the system or execute
commands within the root file system.

12
 Chapt er 2 . File Syst em St ruct ure and Maint enance

/usr/l i bexec

This directory contains small helper programs called by other programs.

/usr/sbi n

As of Red Hat Enterprise Linux 7.0, /sbi n has been moved to /usr/sbi n. This means that
it contains all system administration binaries, including those essential for booting,
restoring, recovering, or repairing the system. The binaries in /usr/sbi n/ require root
privileges to use.

/usr/share

This directory stores files that are not architecture-specific.

/usr/src

This directory stores source code.

/usr/tmp lin ked t o /var/tmp

This directory stores temporary files.

The /usr/ directory should also contain a /l o cal / subdirectory. As per the FHS, this subdirectory
is used by the system administrator when installing software locally, and should be safe from being
overwritten during system updates. The /usr/l o cal directory has a structure similar to /usr/, and
contains the following subdirectories:

/usr/l o cal /bi n

/usr/l o cal /etc

/usr/l o cal /g ames

/usr/l o cal /i ncl ud e

/usr/l o cal /l i b

/usr/l o cal /l i bexec

/usr/l o cal /sbi n

/usr/l o cal /share

/usr/l o cal /src

Red Hat Enterprise Linux's usage of /usr/l o cal / differs slightly from the FHS. The FHS states that
/usr/l o cal / should be used to store softwarethat should remain safe from system software
upgrades. Since the R PM Packag e Man ag er can perform software upgrades safely, it is not
necessary to protect files by storing them in /usr/l o cal /.

Instead, Red Hat Enterprise Linux uses /usr/l o cal / for software local to the machine. For
instance, if the /usr/ directory is mounted as a read-only NFS share from a remote host, it is still
possible to install a package or program under the /usr/l o cal / directory.

2 .1 .1 .1 1 . T he /var/ Dire ct o ry

Since the FHS requires Linux to mount /usr/ as read-only, any programs that write log files or need
spo o l / or l o ck/ directories should write them to the /var/ directory. The FHS states /var/ is for
variable data, which includes spool directories and files, logging data, transient and temporary files.

13
St orage Administ rat ion G uide

Below are some of the directories found within the /var/ directory:

/var/acco unt/

/var/arpwatch/

/var/cache/

/var/crash/

/var/d b/

/var/empty/

/var/ftp/

/var/g d m/

/var/kerbero s/

/var/l i b/

/var/l o cal /

/var/l o ck/

/var/l o g /

/var/mai l linked to /var/spo o l /mai l /

/var/mai l man/

/var/named /

/var/ni s/

/var/o pt/

/var/preserve/

/var/run/

/var/spo o l /

/var/tmp/

/var/tux/

/var/www/

/var/yp/

Important

The /var/run/med i a/user directory contains subdirectories used as mount points for
removable media such as USB storage media, D VD s, CD -ROMs, and Z ip disks. Note that
previously, the /med i a/ directory was used for this purpose.

14
 Chapt er 2 . File Syst em St ruct ure and Maint enance

System log files, such as messag es and l astl o g , go in the /var/l o g / directory. The
/var/l i b/rpm/ directory contains RPM system databases. Lock files go in the /var/l o ck/
directory, usually in directories for the program using the file. The /var/spo o l / directory has
subdirectories that store data files for some programs. These subdirectories include:

/var/spo o l /at/

/var/spo o l /cl i entmq ueue/

/var/spo o l /cro n/

/var/spo o l /cups/

/var/spo o l /exi m/

/var/spo o l /l pd /

/var/spo o l /mai l /

/var/spo o l /mai l man/

/var/spo o l /mq ueue/

/var/spo o l /news/

/var/spo o l /po stfi x/

/var/spo o l /repackag e/

/var/spo o l /rwho /

/var/spo o l /samba/

/var/spo o l /sq ui d /

/var/spo o l /sq ui rrel mai l /

/var/spo o l /up2d ate/

/var/spo o l /uucp/

/var/spo o l /uucppubl i c/

/var/spo o l /vbo x/

2.2. Special Red Hat Ent erprise Linux File Locat ions
Red Hat Enterprise Linux extends the FHS structure slightly to accommodate special files.

Most files pertaining to RPM are kept in the /var/l i b/rpm/ directory. For more information on RPM,
refer to man rpm.

The /var/cache/yum/ directory contains files used by the Packag e U p d at er, including RPM
header information for the system. This location may also be used to temporarily store RPMs
downloaded while updating the system. For more information about the Red Hat Network, refer to the
documentation online at https://rhn.redhat.com/.

Another location specific to Red Hat Enterprise Linux is the /etc/sysco nfi g / directory. This
directory stores a variety of configuration information. Many scripts that run at boot time use the files
in this directory.

15
St orage Administ rat ion G uide

2.3. T he /proc Virt ual File Syst em

Unlike most file systems, /pro c contains neither text nor binary files. Instead, it houses virtual files; as
such, /pro c is normally referred to as a virtual file system. These virtual files are typically zero bytes
in size, even if they contain a large amount of information.

The /pro c file system is not used for storage. Its main purpose is to provide a file-based interface to
hardware, memory, running processes, and other system components. Real-time information can be
retrieved on many system components by viewing the corresponding /pro c file. Some of the files
within /pro c can also be manipulated (by both users and applications) to configure the kernel.

The following /pro c files are relevant in managing and monitoring system storage:

/p ro c/d evices

D isplays various character and block devices that are currently configured.

/p ro c/f ilesyst ems

Lists all file system types currently supported by the kernel.

/p ro c/md st at

Contains current information on multiple-disk or RAID configurations on the system, if they

exist.

/p ro c/mo u n t s

Lists all mounts currently used by the system.

/p ro c/p art it io n s

Contains partition block allocation information.

For more information about the /pro c file system, refer to the Red Hat Enterprise Linux 7 Deployment
Guide.

2.4 . Discard unused blocks

Batch discard and online discard operations are features of mounted file systems that discard blocks
not in use by the file system. They are useful for both solid-state drives and thinly-provisioned
storage.

Batch discard operations are run explicitly by the user with the fstri m command. This command
discards all unused blocks in a file system that match the user's criteria. Both operation types are
supported for use with ext4 file systems as of Red Hat Enterprise Linux 6.2 and later so long as the
block device underlying the file system supports physical discard operations. This is also the case
with XFS file systems as of Red Hat Enterprise Linux 6.4 and later. Physical discard operations are
supported if the value of /sys/bl o ck/device/q ueue/d i scard _max_bytes is not zero.

Online discard operations are specified at mount time with the -o d i scard option (either in
/etc/fstab or as part of the mo unt command), and run in realtime without user intervention. Online
discard operations only discard blocks that are transitioning from used to free. Online discard
operations are supported on ext4 file systems as of Red Hat Enterprise Linux 6.2 and later, and on
XFS file systems as of Red Hat Enterprise Linux 6.4 and later.

Red Hat recommends batch discard operations unless the system's workload is such that batch
discard is not feasible, or online discard operations are necessary to maintain performance.

16
 Chapt er 3. T he XFS File Syst em

Chapter 3. The XFS File System

XFS is a highly scalable, high-performance file system which was originally designed at Silicon
Graphics, Inc. XFS is the default file system for Red Hat Enterprise Linux 7.

Main Feat u res

XFS supports metadata journaling, which facilitates quicker crash recovery. The XFS file
system can also be defragmented and enlarged while mounted and active. In addition, Red
Hat Enterprise Linux 7 supports backup and restore utilities specific to XFS.

Allo cat io n Feat u res

XFS features the following allocation schemes:

Extent-based allocation

Stripe-aware allocation policies

D elayed allocation

Space pre-allocation

D elayed allocation and other performance optimizations affect XFS the same way that they
do ext4. Namely, a program's writes to an XFS file system are not guaranteed to be on-disk
unless the program issues an fsync() call afterwards.

For more information on the implications of delayed allocation on a file system (ext4 and
XFS), refer to Allocation Features in Chapter 5, The Ext4 File System.

Note

Creating or expanding files occasionally fails with an unexpected ENOSPC write

failure even though the disk space appears to be sufficient. This is due to XFS's
performance-oriented design. In practice, it does not become a problem since it only
occurs if remaining space is only a few blocks.

O t h er XFS Feat u res

The XFS file system also supports the following:

Extended attributes ( xattr)

This allows the system to associate several additional name/value pairs per file. It
is enabled by default.

Quota journaling

This avoids the need for lengthy quota consistency checks after a crash.

Project/directory quotas

This allows quota restrictions over a directory tree.

Subsecond timestamps

This allows timestamps to go to the subsecond.

17
St orage Administ rat ion G uide

D ef au lt ati me b eh avio r is rel ati me

R el ati me is on by default for XFS. It has almost no overhead compared to no ati me while
still maintaining sane ati me values.

3.1. Creat ing an XFS File Syst em

To create an XFS file system, use the mkfs. xfs /d ev/device command. In general, the default
options are optimal for common use.

When using mkfs. xfs on a block device containing an existing file system, use the -f option to
force an overwrite of that file system.

Examp le 3.1. mkfs. xfs co mman d o u t p u t

Below is a sample output of the mkfs. xfs command:

meta-data=/dev/device isize=256 agcount=4, agsize=3277258

blks
= sectsz=512 attr=2
data = bsize=4096 blocks=13109032,
imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=6400, version=2
= sectsz=512 sunit=0 blks, lazy-
count=1
realtime =none extsz=4096 blocks=0, rtextents=0

Note

After an XFS file system is created, its size cannot be reduced. However, it can still be enlarged
using the xfs_g ro wfs command (refer to Section 3.4, Increasing the Size of an XFS File
System ).

For striped block devices (for example, RAID 5 arrays), the stripe geometry can be specified at the time
of file system creation. Using proper stripe geometry greatly enhances the performance of an XFS
filesystem.

When creating filesystems on LVM or MD volumes, mkfs. xfs chooses an optimal geometry. This
may also be true on some hardware RAID s that export geometry information to the operating system.

If the device exports stripe geometry information, mkfs (for ext3, ext4, and xfs) will automatically use
this geometry. If stripe geometry is not detected by mkfs and even though the storage does, in fact,
have stripe geometry, it is possible to manually specify it at mkfs time using the following options:

su = value

Specifies a stripe unit or RAID chunk size. The value must be specified in bytes, with an
optional k, m, or g suffix.

sw= value

18
 Chapt er 3. T he XFS File Syst em

Specifies the number of data disks in a RAID device, or the number of stripe units in the
stripe.

The following example specifies a chunk size of 64k on a RAID device containing 4 stripe units:

# mkfs. xfs -d su= 6 4 k,sw= 4 /d ev/device

For more information about creating XFS file systems, refer to man mkfs. xfs and the Red Hat
Enterprise Linux Performance Tuning Guide, chapter Basic Tuning for XFS.

3.2. Mount ing an XFS File Syst em

An XFS file system can be mounted with no extra options, for example:

# mo unt /d ev/device /mount/point

The default for Red Hat Enterprise Linux 7 is inode64.

Note

Unlike mke2fs, mkfs.xfs does not utilize a configuration file; they are all specified on the
command line.

Writ e Barriers

By default, XFS uses write barriers to ensure file system integrity even when power is lost to a device
with write caches enabled. For devices without write caches, or with battery-backed write caches,
disable the barriers by using the no barri er option:

# mo unt -o no barri er /dev/device /mount/point

For more information about write barriers, refer to Chapter 21, Write Barriers.

Direct Access T echnology Preview

Starting with Red Hat Enterprise Linux 7.3, D i rect Access (D AX) provides, as a Technology
Preview on the ext4 and XFS file systems, a means for an application to directly map persistent
memory into its address space. To use D AX, a system must have some form of persistent memory
available, usually in the form of one or more Non-Volatile D ual In-line Memory Modules (NVD IMMs),
and a file system that supports D AX must be created on the NVD IMM(s). Also, the file system must be
mounted with the d ax mount option. Then, an mmap of a file on the dax-mounted file system results in
a direct mapping of storage into the application's address space.

3.3. XFS Quot a Management

The XFS quota subsystem manages limits on disk space (blocks) and file (inode) usage. XFS quotas
control or report on usage of these items on a user, group, or directory or project level. Also, note that
while user, group, and directory or project quotas are enabled independently, group and project
quotas are mutually exclusive.

19
St orage Administ rat ion G uide

When managing on a per-directory or per-project basis, XFS manages the disk usage of directory
hierarchies associated with a specific project. In doing so, XFS recognizes cross-organizational
" group" boundaries between projects. This provides a level of control that is broader than what is
available when managing quotas for users or groups.

XFS quotas are enabled at mount time, with specific mount options. Each mount option can also be
specified as no enfo rce; this will allow usage reporting without enforcing any limits. Valid quota
mount options are:

uq uo ta/uq no enfo rce - User quotas

g q uo ta/g q no enfo rce - Group quotas

pq uo ta/pq no enfo rce - Project quota

Once quotas are enabled, the xfs_q uo ta tool can be used to set limits and report on disk usage. By
default, xfs_q uo ta is run interactively, and in basic mode. Basic mode sub-commands simply report
usage, and are available to all users. Basic xfs_q uo ta sub-commands include:

q u o t a username/userID

Show usage and limits for the given username or numeric userID

df

Shows free and used counts for blocks and inodes.

In contrast, xfs_q uo ta also has an expert mode. The sub-commands of this mode allow actual
configuration of limits, and are available only to users with elevated privileges. To use expert mode
sub-commands interactively, run xfs_q uo ta -x. Expert mode sub-commands include:

rep o rt /path

Reports quota information for a specific file system.

limit

Modify quota limits.

For a complete list of sub-commands for either basic or expert mode, use the sub-command hel p.

All sub-commands can also be run directly from a command line using the -c option, with -x for
expert sub-commands.

Examp le 3.2. D isp lay a samp le q u o t a rep o rt

For example, to display a sample quota report for /ho me (on /d ev/bl o ckd evi ce), use the
command xfs_q uo ta -x -c ' repo rt -h' /ho me. This will display output similar to the
following:

User quota on /home (/dev/blockdevice)

Blocks
User ID Used Soft Hard Warn/Grace
---------- ---------------------------------
root 0 0 0 00 [------]
testuser 103.4G 0 0 00 [------]
...

20
 Chapt er 3. T he XFS File Syst em

To set a soft and hard inode count limit of 500 and 700 respectively for user jo hn (whose home
directory is /ho me/jo hn), use the following command:

# xfs_q uo ta -x -c ' l i mi t i so ft= 50 0 i hard = 70 0 jo hn' /ho me/

In this case, pass mount_point which is the mounted xfs file system.

By default, the l i mi t sub-command recognizes targets as users. When configuring the limits for a
group, use the -g option (as in the previous example). Similarly, use -p for projects.

Soft and hard block limits can also be configured using bso ft or bhard instead of i so ft or
i hard .

Examp le 3.3. Set a so f t an d h ard b lo ck limit

For example, to set a soft and hard block limit of 1000m and 1200m, respectively, to group
acco unti ng on the /targ et/path file system, use the following command:

# xfs_q uo ta -x -c ' l i mi t -g bso ft= 10 0 0 m bhard = 120 0 m acco unti ng '

/targ et/path

Note

The commands bso ft and bhard count by the byte.

Important

While real-time blocks (rtbhard /rtbso ft) are described in man xfs_q uo ta as valid units
when setting quotas, the real-time sub-volume is not enabled in this release. As such, the
rtbhard and rtbso ft options are not applicable.

Set t ing Project Limit s

Before configuring limits for project-controlled directories, add them first to /etc/pro jects. Project
names can be added to /etc/pro jecti d to map project ID s to project names. Once a project is
added to /etc/pro jects, initialize its project directory using the following command:

# xfs_q uo ta -x -c ' pro ject -s projectname' project_path

Quotas for projects with initialized directories can then be configured, with:

# xfs_q uo ta -x -c ' l i mi t -p bso ft= 10 0 0 m bhard = 120 0 m projectname'

Generic quota configuration tools (q uo ta, repq uo ta, and ed q uo ta for example) may also be used
to manipulate XFS quotas. However, these tools cannot be used with XFS project quotas.

21
St orage Administ rat ion G uide

Important

Red Hat recommends the use of xfs_q uo ta over all other available tools.

For more information about setting XFS quotas, refer to man xfs_q uo ta, man pro ji d (5), and
man pro jects(5).

3.4 . Increasing t he Siz e of an XFS File Syst em

An XFS file system may be grown while mounted using the xfs_g ro wfs command:

# xfs_g ro wfs /mount/point -D size

The -D size option grows the file system to the specified size (expressed in file system blocks).
Without the -D size option, xfs_g ro wfs will grow the file system to the maximum size supported
by the device.

Before growing an XFS file system with -D size, ensure that the underlying block device is of an
appropriate size to hold the file system later. Use the appropriate resizing methods for the affected
block device.

Note

While XFS file systems can be grown while mounted, their size cannot be reduced at all.

For more information about growing a file system, refer to man xfs_g ro wfs.

3.5. Repairing an XFS File Syst em

To repair an XFS file system, use xfs_repai r:

# xfs_repai r /dev/device

The xfs_repai r utility is highly scalable and is designed to repair even very large file systems with
many inodes efficiently. Unlike other Linux file systems, xfs_repai r does not run at boot time, even
when an XFS file system was not cleanly unmounted. In the event of an unclean unmount,
xfs_repai r simply replays the log at mount time, ensuring a consistent file system.

Warning

The xfs_repai r utility cannot repair an XFS file system with a dirty log. To clear the log,
mount and unmount the XFS file system. If the log is corrupt and cannot be replayed, use the -
L option (" force log zeroing" ) to clear the log, that is, xfs_repai r -L /dev/device. Be
aware that this may result in further corruption or data loss.

For more information about repairing an XFS file system, refer to man xfs_repai r.

22
 Chapt er 3. T he XFS File Syst em

3.6. Suspending an XFS File Syst em

To suspend or resume write activity to a file system, use xfs_freeze. Suspending write activity
allows hardware-based device snapshots to be used to capture the file system in a consistent state.

Note

The xfs_freeze utility is provided by the xfspro g s package, which is only available on
x86_64.

To suspend (that is, freeze) an XFS file system, use:

# xfs_freeze -f /mount/point

To unfreeze an XFS file system, use:

# xfs_freeze -u /mount/point

When taking an LVM snapshot, it is not necessary to use xfs_freeze to suspend the file system
first. Rather, the LVM management tools will automatically suspend the XFS file system before taking
the snapshot.

For more information about freezing and unfreezing an XFS file system, refer to man xfs_freeze.

3.7. Backup and Rest orat ion of XFS File Syst ems

XFS file system backup and restoration involves two utilities: xfsd ump and xfsresto re.

To backup or dump an XFS file system, use the xfsd ump utility. Red Hat Enterprise Linux 7 supports
backups to tape drives or regular file images, and also allows multiple dumps to be written to the
same tape. The xfsd ump utility also allows a dump to span multiple tapes, although only one dump
can be written to a regular file. In addition, xfsd ump supports incremental backups, and can exclude
files from a backup using size, subtree, or inode flags to filter them.

In order to support incremental backups, xfsd ump uses dump levels to determine a base dump to
which a specific dump is relative. The -l option specifies a dump level (0-9). To perform a full
backup, perform a level 0 dump on the file system (that is, /path/to/filesystem), as in:

# xfsd ump -l 0 -f /d ev/device /path/to/filesystem

Note

The -f option specifies a destination for a backup. For example, the /d ev/st0 destination is
normally used for tape drives. An xfsd ump destination can be a tape drive, regular file, or
remote tape device.

In contrast, an incremental backup will only dump files that changed since the last level 0 dump. A
level 1 dump is the first incremental dump after a full dump; the next incremental dump would be level
2, and so on, to a maximum of level 9. So, to perform a level 1 dump to a tape drive:

23
St orage Administ rat ion G uide

# xfsd ump -l 1 -f /d ev/st0 /path/to/filesystem

Conversely, the xfsresto re utility restores file systems from dumps produced by xfsd ump. The
xfsresto re utility has two modes: a default simple mode, and a cumulative mode. Specific dumps are
identified by session ID or session label. As such, restoring a dump requires its corresponding session
ID or label. To display the session ID and labels of all dumps (both full and incremental), use the -I
option:

# xfsresto re -I

This will provide output similar to the following:

Examp le 3.4 . Sessio n ID an d lab els o f all d u mp s

file system 0:
fs id: 45e9af35-efd2-4244-87bc-4762e476cbab
session 0:
mount point: bear-05:/mnt/test
device: bear-05:/dev/sdb2
time: Fri Feb 26 16:55:21 2010
session label: "my_dump_session_label"
session id: b74a3586-e52e-4a4a-8775-c3334fa8ea2c
level: 0
resumed: NO
subtree: NO
streams: 1
stream 0:
pathname: /mnt/test2/backup
start: ino 0 offset 0
end: ino 1 offset 0
interrupted: NO
media files: 1
media file 0:
mfile index: 0
mfile type: data
mfile size: 21016
mfile start: ino 0 offset 0
mfile end: ino 1 offset 0
media label: "my_dump_media_label"
media id: 4a518062-2a8f-4f17-81fd-bb1eb2e3cb4f
xfsrestore: Restore Status: SUCCESS

Simple Mode for xfsresto re

The simple mode allows users to restore an entire file system from a level 0 dump. After identifying a
level 0 dump's session ID (that is, session-ID), restore it fully to /path/to/destination using:

# xfsresto re -f /d ev/st0 -S session-ID /path/to/destination

24
 Chapt er 3. T he XFS File Syst em

Note

The -f option specifies the location of the dump, while the -S or -L option specifies which
specific dump to restore. The -S option is used to specify a session ID , while the -L option is
used for session labels. The -I option displays both session labels and ID s for each dump.

Cumulat ive Mode for xfsresto re

The cumulative mode of xfsresto re allows file system restoration from a specific incremental
backup, for example, level 1 to level 9. To restore a file system from an incremental backup, simply add
the -r option:

# xfsresto re -f /d ev/st0 -S session-ID -r /path/to/destination

Int eract ive Operat ion

The xfsresto re utility also allows specific files from a dump to be extracted, added, or deleted. To
use xfsresto re interactively, use the -i option, as in:

xfsresto re -f /d ev/st0 -i /destination/directory

The interactive dialogue will begin after xfsresto re finishes reading the specified device. Available
commands in this dialogue include cd , l s, ad d , d el ete, and extract; for a complete list of
commands, use hel p.

Informat ional Messages When Rest oring a Backup from a T ape

When restoring a backup from a tape with backups from multiple file systems, the xfsresto re utility
might issue messages. The messages inform you whether a match of the requested backup has been
found when xfsresto re examines each backup on the tape in sequential order. For example:

xfsrestore: preparing drive

xfsrestore: examining media file 0
xfsrestore: inventory session uuid (8590224e-3c93-469c-a311-fc8f23029b2a)
does not match the media header's session uuid (7eda9f86-f1e9-4dfd-b1d4-
c50467912408)
xfsrestore: examining media file 1
xfsrestore: inventory session uuid (8590224e-3c93-469c-a311-fc8f23029b2a)
does not match the media header's session uuid (7eda9f86-f1e9-4dfd-b1d4-
c50467912408)
[...]

The informational messages keep appearing until the matching backup is found.

For more information about dumping and restoring XFS file systems, refer to man xfsd ump and man
xfsresto re.

3.8. Ot her XFS File Syst em Ut ilit ies

Red Hat Enterprise Linux 7 also features other utilities for managing XFS file systems:

xf s_f sr

25
St orage Administ rat ion G uide

Used to defragment mounted XFS file systems. When invoked with no arguments, xfs_fsr
defragments all regular files in all mounted XFS file systems. This utility also allows users to
suspend a defragmentation at a specified time and resume from where it left off later.

In addition, xfs_fsr also allows the defragmentation of only one file, as in xfs_fsr
/path/to/file. Red Hat advises not to periodically defrag an entire file system because
XFS avoids fragmentation by default. System wide defragmentation could cause the side
effect of fragmentation in free space.

xf s_b map

Prints the map of disk blocks used by files in an XFS filesystem. This map lists each extent
used by a specified file, as well as regions in the file with no corresponding blocks (that is,
holes).

xf s_in f o

Prints XFS file system information.

xf s_ad min

Changes the parameters of an XFS file system. The xfs_ad mi n utility can only modify
parameters of unmounted devices or file systems.

xf s_co p y

Copies the contents of an entire XFS file system to one or more targets in parallel.

The following utilities are also useful in debugging and analyzing XFS file systems:

xf s_met ad u mp

Copies XFS file system metadata to a file. Red Hat only supports using the xfs_metad ump
utility to copy unmounted file systems or read-only mounted file systems; otherwise,
generated dumps could be corrupted or inconsistent.

xf s_md rest o re

Restores an XFS metadump image (generated using xfs_metad ump) to a file system
image.

xf s_d b

D ebugs an XFS file system.

For more information about these utilities, refer to their respective man pages.

3.9. Migrat ing from ext 4 t o XFS

Starting with Red Hat Enterprise Linux 7.0, XFS is the default file system instead of ext4. This section
highlights the differences when using or administering an XFS file system.

The ext4 file system is still fully supported in Red Hat Enterprise Linux 7 and can be selected at
installation. While it is possible to migrate from ext4 to XFS, it is not required.

3.9.1. Commands used wit h ext 3 and ext 4 compared t o XFS

26
 Chapt er 3. T he XFS File Syst em

The following table compares common commands used with ext3 and ext4 to their XFS-specific
counterparts.

T ab le 3.1. C o mmo n C o mman d s f o r ext 3 an d ext 4 C o mp ared t o XFS

T ask ext 3/4 XFS

Create a file system mkfs. ext4 or mkfs. ext3 mkfs. xfs
File system check e2fsck xfs_repai r
Resizing a file system resi ze2fs xfs_g ro wfs
Save an image of a file system e2i mag e xfs_metad ump and
xfs_md resto re
Label or tune a file system tune2fs xfs_ad mi n
Backup a file system d ump and resto re xfsd ump and xfsresto re

The following table lists generic tools that function on XFS file systems as well, but the XFS versions
have more specific functionality and as such are recommended.

T ab le 3.2. G en eric t o o ls f o r ext 4 an d XFS

T ask ext 4 XFS

Quota q uo ta xfs_q uo ta
File mapping fi l efrag xfs_bmap

More information on many the listed XFS commands is included in Chapter 3, The XFS File System.
You can also consult the manual pages of the listed XFS administration tools for more information.

3.9.2. Behavioral and Administ rat ive Differences Bet ween Ext 3/4 and XFS

File syst em rep air

Ext3/4 runs e2fsck in userspace at boot time to recover the journal as needed. XFS, by
comparison, performs journal recovery in kernelspace at mount time. An fsck. xfs shell
script is provided but does not perform any useful action as it is only there to satisfy
initscript requirements.

When an XFS file system repair or check is requested, use the xfs_repai r command. Use
the -n option for a read-only check.

The xfs_repai r command will not operate on a file system with a dirty log. To repair such
a file system mo unt and unmo unt must first be performed to replay the log. If the log is
corrupt and cannot be replayed, the -L option can be used to zero out in the log.

For more information on file system repair of XFS file systems, refer to Section 11.2.2, XFS

Met ad at a erro r b eh avio r

The ext3/4 file system has configurable behavior when metadata errors are encountered,
with the default being to simply continue. When XFS encounters a metadata error that is not
recoverable it will shut down the file system and return a EFSC O R R UP T ED error. The system
logs will contain details of the error enountered and will recommend running xfs_repai r if
necessary.

Q u o t as

XFS quotas are not a remountable option. The -o q uo ta option must be specified on the
initial mount for quotas to be in effect.

27
St orage Administ rat ion G uide

While the standard tools in the quota package can perform basic quota administrative
tasks (tools such as set q u o t a and rep q u o t a), the xf s_q u o t a tool can be used for XFS-
specific features, such as Project Quota administration.

The q uo tacheck command has no effect on an XFS file system. The first time quota
accounting is turned on XFS does an automatic q uo tacheck internally. Because XFS
quota metadata is a first-class, journaled metadata object, the quota system will always be
consistent until quotas are manually turned off.

File syst em resiz e

The XFS file system has no utility to shrink a file system. XFS file systems can be grown
online via the xfs_g ro wfs command.

In o d e n u mb ers

For file systems larger than 1 TB with 256-byte inodes, or larger than 2 TB with 512-byte
inodes, XFS inode numbers might exceed 2^32. Such large inode numbers cause 32-bit
stat calls to fail with the EOVERFLOW return value. The described problem might occur
when using the default Red Hat Enterprise Linux 7 configuration: non-striped with four
allocation groups. A custom configuration, for example file system extension or changing
XFS file system parameters, might lead to a different behavior.

Applications usually handle such larger inode numbers correctly. If needed, mount the XFS
file system with the -o i no d e32 parameter to enforce inode numbers below 2^32. Note
that using i no d e32 does not affect inodes that are already allocated with 64-bit numbers.

Important

D o not use the i no d e32 option unless it is required by a specific environment. The
i no d e32 option changes allocation behavior. As a consequence, the ENOSPC
error might occur if no space is available to allocate inodes in the lower disk blocks.

Sp ecu lat ive p reallo cat io n

XFS uses speculative preallocation to allocate blocks past EOF as files are written. This
avoids file fragmentation due to concurrent streaming write workloads on NFS servers. By
default, this preallocation increases with the size of the file and will be apparent in " du"
output. If a file with speculative preallocation is not dirtied for five minutes the preallocation
will be discarded. If the inode is cycled out of cache before that time, then the preallocation
will be discarded when the inode is reclaimed.

If premature ENOSPC problems are seen due to speculative preallocation, a fixed

preallocation amount may be specified with the -o al l o csi ze= amount mount option.

Frag men t at io n - relat ed t o o ls

Fragmentation is rarely a significant issue on XFS file systems due to heuristics and
behaviors, such as delayed allocation and speculative preallocation. However, tools exist
for measuring file system fragmentation as well as defragmenting file systems. Their use is
not encouraged.

The xfs_d b frag command attempts to distill all file system allocations into a single
fragmentation number, expressed as a percentage. The output of the comand requires
significant expertise to understand its meaning. For example, a fragmentation factor of 75%
means only an average of 4 extents per file. For this reason the output of xfs_db's frag is
not considered useful and more careful analysis of any fragmentation problems is

28
 Chapt er 3. T he XFS File Syst em

recommended.

Warning

The xfs_fsr command may be used to defragment individual files, or all files on a
file system. The later is especially not recommended as it may destroy locality of files
and may fragment freespace.

29
St orage Administ rat ion G uide

Chapter 4. The Ext3 File System

The ext3 file system is essentially an enhanced version of the ext2 file system. These improvements
provide the following advantages:

Availab ilit y

After an unexpected power failure or system crash (also called an unclean system shutdown),
each mounted ext2 file system on the machine must be checked for consistency by the
e2fsck program. This is a time-consuming process that can delay system boot time
significantly, especially with large volumes containing a large number of files. D uring this
time, any data on the volumes is unreachable.

It is possible to run fsck -n on a live filesystem. However, it will not make any changes
and may give misleading results if partially written metadata is encountered.

If LVM is used in the stack, another option is to take an LVM snapshot of the filesystem and
run fsck on it instead.

Finally, there is the option to remount the filesystem as read only. All pending metadata
updates (and writes) are then forced to the disk prior to the remount. This ensures the
filesystem is in a consistent state, provided there is no previous corruption. It is now
possible to run fsck -n.

The journaling provided by the ext3 file system means that this sort of file system check is
no longer necessary after an unclean system shutdown. The only time a consistency check
occurs using ext3 is in certain rare hardware failure cases, such as hard drive failures. The
time to recover an ext3 file system after an unclean system shutdown does not depend on
the size of the file system or the number of files; rather, it depends on the size of the journal
used to maintain consistency. The default journal size takes about a second to recover,
depending on the speed of the hardware.

Note

The only journaling mode in ext3 supported by Red Hat is d ata= o rd ered (default).

D at a In t eg rit y

The ext3 file system prevents loss of data integrity in the event that an unclean system
shutdown occurs. The ext3 file system allows you to choose the type and level of protection
that your data receives. With regard to the state of the file system, ext3 volumes are
configured to keep a high level of data consistency by default.

Sp eed

D espite writing some data more than once, ext3 has a higher throughput in most cases
than ext2 because ext3's journaling optimizes hard drive head motion. You can choose
from three journaling modes to optimize speed, but doing so means trade-offs in regards to
data integrity if the system was to fail.

Note

The only journaling mode in ext3 supported by Red Hat is d ata= o rd ered (default).

30
 Chapt er 4 . T he Ext 3 File Syst em

Easy T ran sit io n

It is easy to migrate from ext2 to ext3 and gain the benefits of a robust journaling file system
without reformatting. Refer to Section 4.2, Converting to an Ext3 File System for more
information on how to perform this task.

Note

Red Hat Enterprise Linux 7 provides a unified extN driver. It does this by disabling the ext2 and
ext3 configurations and instead uses ext4 . ko for these on-disk formats. This means that
kernel messages will always refer to ext4 regardless of the ext file system used.

The following sections cover creating and tuning ext3 partitions. For ext2 partitions, skip the
partitioning and formatting sections below and go directly to Section 4.2, Converting to an Ext3 File
System .

4 .1. Creat ing an Ext 3 File Syst em

After installation, it is sometimes necessary to create a new ext3 file system. For example, if a new disk
drive is added to the system, you may want to partition the drive and use the ext3 file system.

The steps for creating an ext3 file system are as follows:

Pro ced u re 4 .1. C reat e an ext 3 f ile syst em

1. Format the partition or LVM volume with the ext3 file system using mkfs.

2. Label the file system using e2l abel .

It is also possible to add a specific UUID to a file system. For example, to add the UUID 7cd65de3-
e0be-41d9-b66d-96d749c02da7 to the file system /d ev/sd a8, use the following commands:

# mkfs -t ext3 -U 7cd65de3-e0be-41d9-b66d-96d749c02da7 /dev/sda8

# tune2fs -U 7cd65de3-e0be-41d9-b66d-96d749c02da7 /dev/sda8

Replace the ext3 with the file system type you are using (ext4, for example), followed by the -U option
with your chosen UUID , and replace the /dev/sda8 with the file system to have the UUID added to it.

4 .2. Convert ing t o an Ext 3 File Syst em

The tune2fs command converts an ext2 file system to ext3.

Note

To convert ext2 to ext3, always use the e2fsck utility to check your file system before and after
using tune2fs. Before trying to convert ext2 to ext3, back up all file systems in case any
errors occur.

In addition, Red Hat recommends creating a new ext3 file system and migrating data to it,
instead of converting from ext2 to ext3 whenever possible.

31
St orage Administ rat ion G uide

To convert an ext2 file system to ext3, log in as root and type the following command in a terminal:

# tune2fs -j block_device

block_device contains the ext2 file system to be converted.

Issue the d f command to display mounted file systems.

4 .3. Revert ing t o an Ext 2 File Syst em

In order to revert to an ext2 file system, use the following procedure.

For simplicity, the sample commands in this section use the following value for the block device:

/d ev/mapper/Vo l G ro up0 0 -Lo g Vo l 0 2

Pro ced u re 4 .2. R evert f ro m ext 3 t o ext 2

1. Unmount the partition by logging in as root and typing:

# umo unt /dev/mapper/VolGroup00-LogVol02

2. Change the file system type to ext2 by typing the following command:

# tune2fs -O ^has_jo urnal /dev/mapper/VolGroup00-LogVol02

3. Check the partition for errors by typing the following command:

# e2fsck -y /dev/mapper/VolGroup00-LogVol02

4. Then mount the partition again as ext2 file system by typing:

# mo unt -t ext2 /dev/mapper/VolGroup00-LogVol02 /mount/point

In the above command, replace /mount/point with the mount point of the partition.

Note

If a . jo urnal file exists at the root level of the partition, delete it.

To permanently change the partition to ext2, remember to update the /etc/fstab file, otherwise it
will revert back after booting.

32
 Chapt er 5. T he Ext 4 File Syst em

Chapter 5. The Ext4 File System

The ext4 file system is a scalable extension of the ext3 file system. With Red Hat Enterprise Linux 7, it
can support a maximum individual file size of 16 terabytes, and file systems to a maximum of 50
terabytes, unlike Red Hat Enterprise Linux 6 which only supported file systems up to 16 terabytes. It
also supports an unlimited number of sub-directories (the ext3 file system only supports up to
32,000), though once the link count exceeds 65,000 it resets to 1 and is no longer increased. The
bigalloc feature is not currently supported.

Note

As with ext3, an ext4 volume must be umounted in order to perform an fsck. For more
information, see Chapter 4, The Ext3 File System.

Main Feat u res

Ext4 uses extents (as opposed to the traditional block mapping scheme used by ext2 and
ext3), which improves performance when using large files and reduces metadata overhead
for large files. In addition, ext4 also labels unallocated block groups and inode table
sections accordingly, which allows them to be skipped during a file system check. This
makes for quicker file system checks, which becomes more beneficial as the file system
grows in size.

Allo cat io n Feat u res

The ext4 file system features the following allocation schemes:

Persistent pre-allocation

D elayed allocation

Multi-block allocation

Stripe-aware allocation

Because of delayed allocation and other performance optimizations, ext4's behavior of

writing files to disk is different from ext3. In ext4, when a program writes to the file system, it
is not guaranteed to be on-disk unless the program issues an fsync() call afterwards.

By default, ext3 automatically forces newly created files to disk almost immediately even
without fsync(). This behavior hid bugs in programs that did not use fsync() to ensure
that written data was on-disk. The ext4 file system, on the other hand, often waits several
seconds to write out changes to disk, allowing it to combine and reorder writes for better
disk performance than ext3.

Warning

Unlike ext3, the ext4 file system does not force data to disk on transaction commit. As
such, it takes longer for buffered writes to be flushed to disk. As with any file system,
use data integrity calls such as fsync() to ensure that data is written to permanent
storage.

O t h er Ext 4 Feat u res

33
St orage Administ rat ion G uide

The ext4 file system also supports the following:

Extended attributes (xattr) This allows the system to associate several additional
name and value pairs per file.

Quota journaling This avoids the need for lengthy quota consistency checks after a
crash.

Note

The only supported journaling mode in ext4 is d ata= o rd ered (default).

Subsecond timestamps This gives timestamps to the subsecond.

5.1. Creat ing an Ext 4 File Syst em

To create an ext4 file system, use the mkfs. ext4 command. In general, the default options are
optimal for most usage scenarios:

# mkfs. ext4 /d ev/device

Below is a sample output of this command, which displays the resulting file system geometry and
features:

Examp le 5.1. mkfs. ext4 co mman d o u t p u t

~]# mkfs.ext4 /dev/sdb1

mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
245280 inodes, 979456 blocks
48972 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1006632960
30 block groups
32768 blocks per group, 32768 fragments per group
8176 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done

Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

For striped block devices (for example, RAID 5 arrays), the stripe geometry can be specified at the time
of file system creation. Using proper stripe geometry greatly enhances the performance of an ext4 file
system.

34
 Chapt er 5. T he Ext 4 File Syst em

When creating file systems on LVM or MD volumes, mkfs. ext4 chooses an optimal geometry. This
may also be true on some hardware RAID s which export geometry information to the operating
system.

To specify stripe geometry, use the -E option of mkfs. ext4 (that is, extended file system options)
with the following sub-options:

st rid e= value

Specifies the RAID chunk size.

st rip e- wid t h = value

Specifies the number of data disks in a RAID device, or the number of stripe units in the
stripe.

For both sub-options, value must be specified in file system block units. For example, to create a file
system with a 64k stride (that is, 16 x 4096) on a 4k-block file system, use the following command:

# mkfs. ext4 -E stri d e= 16 ,stri pe-wi d th= 6 4 /d ev/device

For more information about creating file systems, refer to man mkfs. ext4 .

Important

It is possible to use tune2fs to enable certain ext4 features on ext3 file systems. However,
using tune2fs in this way has not been fully tested and is therefore not supported in Red Hat
Enterprise Linux 7. As a result, Red Hat cannot guarantee consistent performance and
predictable behavior for ext3 file systems converted or mounted by using tune2fs.

It is also possible to add a specific UUID to the file system. See Section 4.1, Creating an Ext3 File
System for more information.

5.2. Mount ing an Ext 4 File Syst em

An ext4 file system can be mounted with no extra options. For example:

# mo unt /d ev/device /mount/point

The ext4 file system also supports several mount options to influence behavior. For example, the acl
parameter enables access control lists, while the user_xattr parameter enables user extended
attributes. To enable both options, use their respective parameters with -o , as in:

# mo unt -o acl ,user_xattr /d ev/device /mount/point

As with ext3, the option d ata_err= abo rt can be used to abort the journal if an error occures in file
data.

# mount -o data_err=abort /dev/device /mount/point

The tune2fs utility also allows administrators to set default mount options in the file system
superblock. For more information on this, refer to man tune2fs.

35
St orage Administ rat ion G uide

Writ e Barriers

By default, ext4 uses write barriers to ensure file system integrity even when power is lost to a device
with write caches enabled. For devices without write caches, or with battery-backed write caches,
disable barriers using the no barri er option, as in:

# mo unt -o no barri er /d ev/device /mount/point

For more information about write barriers, refer to Chapter 21, Write Barriers.

Direct Access T echnology Preview

Starting with Red Hat Enterprise Linux 7.3, D i rect Access (D AX) provides, as a Technology
Preview on the ext4 and XFS file systems, a means for an application to directly map persistent
memory into its address space. To use D AX, a system must have some form of persistent memory
available, usually in the form of one or more Non-Volatile D ual In-line Memory Modules (NVD IMMs),
and a file system that supports D AX must be created on the NVD IMM(s). Also, the file system must be
mounted with the d ax mount option. Then, an mmap of a file on the dax-mounted file system results in
a direct mapping of storage into the application's address space.

5.3. Resiz ing an Ext 4 File Syst em

Before growing an ext4 file system, ensure that the underlying block device is of an appropriate size
to hold the file system later. Use the appropriate resizing methods for the affected block device.

An ext4 file system may be grown while mounted using the resi ze2fs command:

# resi ze2fs /mount/device size

The resi ze2fs command can also decrease the size of an unmounted ext4 file system:

# resi ze2fs /d ev/device size

When resizing an ext4 file system, the resi ze2fs utility reads the size in units of file system block
size, unless a suffix indicating a specific unit is used. The following suffixes indicate specific units:

s 512 byte sectors

K kilobytes

M megabytes

G gigabytes

Note

The size parameter is optional (and often redundant) when expanding. The resi ze2fs
automatically expands to fill all available space of the container, usually a logical volume or
partition.

For more information about resizing an ext4 file system, refer to man resi ze2fs.

36
 Chapt er 5. T he Ext 4 File Syst em

5.4 . Backup ext 2/3/4 File Syst ems

Pro ced u re 5.1. B acku p ext 2/3/4 File Syst ems Examp le

1. All data must be backed up before attempting any kind of restore operation. D ata backups
should be made on a regular basis. In addition to data, there is configuration information
that should be saved, including /etc/fstab and the output of fd i sk -l . Running an
sosreport/sysreport will capture this information and is strongly recommended.

# cat /etc/fstab
LABEL=/ / ext3 defaults 1 1
LABEL=/boot1 /boot ext3 defaults 1 2
LABEL=/data /data ext3 defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda5 swap swap defaults 0 0
/dev/sda6 /backup-files ext3 defaults 0 0

# fdisk -l
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1925 15358140 83 Linux
/dev/sda3 1926 3200 10241437+ 83 Linux
/dev/sda4 3201 4864 13366080 5 Extended
/dev/sda5 3201 3391 1534176 82 Linux swap /
Solaris
/dev/sda6 3392 4864 11831841 83 Linux

In this example, we will use the /d ev/sd a6 partition to save backup files, and we assume
that /d ev/sd a6 is mounted on /backup-fi l es.

2. If the partition being backed up is an operating system partition, bootup your system into
Single User Mode. This step is not necessary for normal data partitions.

3. Use d ump to backup the contents of the partitions:

37
St orage Administ rat ion G uide

Note

If the system has been running for a long time, it is advisable to run e2fsck on the
partitions before backup.
d ump should not be used on heavily loaded and mounted filesystem as it could
backup corrupted version of files. This problem has been mentioned on
dump.sourceforge.net.

Important

When backing up operating system partitions, the partition must be

unmounted.

While it is possible to back up an ordinary data partition while it is mounted,

it is adviseable to unmount it where possible. The results of attempting to
back up a mounted data partition can be unpredicteable.

# dump -0uf /backup-files/sda1.dump /dev/sda1

# dump -0uf /backup-files/sda2.dump /dev/sda2
# dump -0uf /backup-files/sda3.dump /dev/sda3

If you want to do a remote backup, you can use both ssh or configure a non-password login.

Note

If using standard redirection, the '-f' option must be passed separately.

# dump -0u -f - /dev/sda1 | ssh root@ remoteserver.example.com dd

of=/tmp/sda1.dump

5.5. Rest ore an ext 2/3/4 File Syst em

Pro ced u re 5.2. R est o re an ext 2/3/4 File Syst em Examp le

1. If you are restoring an operating system partition, bootup your system into Rescue Mode.
This step is not required for ordinary data partitions.

2. Rebuild sda1/sda2/sda3/sda4/sda5 by using the fd i sk command.

38
 Chapt er 5. T he Ext 4 File Syst em

Note

If necessary, create the partitions to contain the restored file systems. The new
partitions must be large enough to contain the restored data. It is important to get the
start and end numbers right; these are the starting and ending sector numbers of the
partitions.

3. Format the destination partitions by using the mkfs command, as shown below.

Important

D O NOT format /d ev/sd a6 in the above example because it saves backup files.

# mkfs.ext3 /dev/sda1
# mkfs.ext3 /dev/sda2
# mkfs.ext3 /dev/sda3

4. If creating new partitions, re-label all the partitions so they match the fstab file. This step is
not required if the partitions are not being recreated.

# e2label /dev/sda1 /boot1

# e2label /dev/sda2 /
# e2label /dev/sda3 /data
# mkswap -L SWAP-sda5 /dev/sda5

5. Prepare the working directories.

# mkdir /mnt/sda1
# mount -t ext3 /dev/sda1 /mnt/sda1
# mkdir /mnt/sda2
# mount -t ext3 /dev/sda2 /mnt/sda2
# mkdir /mnt/sda3
# mount -t ext3 /dev/sda3 /mnt/sda3
# mkdir /backup-files
# mount -t ext3 /dev/sda6 /backup-files

6. Restore the data.

# cd /mnt/sda1
# restore -rf /backup-files/sda1.dump
# cd /mnt/sda2
# restore -rf /backup-files/sda2.dump
# cd /mnt/sda3
# restore -rf /backup-files/sda3.dump

If you want to restore from a remote host or restore from a backup file on a remote host you
can use either ssh or rsh. You will need to configure a password-less login for the following
examples:

Login into 10.0.0.87, and restore sda1 from local sda1.dump file:

39
St orage Administ rat ion G uide

# ssh 10.0.0.87 "cd /mnt/sda1 & & cat /backup-files/sda1.dump |

restore -rf -"

Login into 10.0.0.87, and restore sda1 from a remote 10.66.0.124 sda1.dump file:

# ssh 10.0.0.87 "cd /mnt/sda1 & & RSH=/usr/bin/ssh restore -r -f

10.66.0.124:/tmp/sda1.dump"

7. Reboot.

5.6. Ot her Ext 4 File Syst em Ut ilit ies

Red Hat Enterprise Linux 7 also features other utilities for managing ext4 file systems:

e2f sck

Used to repair an ext4 file system. This tool checks and repairs an ext4 file system more
efficiently than ext3, thanks to updates in the ext4 disk structure.

e2lab el

Changes the label on an ext4 file system. This tool also works on ext2 and ext3 file systems.

quota

Controls and reports on disk space (blocks) and file (inode) usage by users and groups on
an ext4 file system. For more information on using q uo ta, refer to man q uo ta and
Section 16.1, Configuring D isk Quotas .

f sf reez e

To suspend access to a file system, use the command # fsfreeze -f mount-point to

freeze it and # fsfreeze -u mount-point to unfreeze it. This halts access to the file
system and creates a stable image on disk.

Note

It is unnecessary to use fsfreeze for device-mapper drives.

For more information see the fsfreeze(8) manpage.

As demonstrated in Section 5.2, Mounting an Ext4 File System , the tune2fs utility can also adjust
configurable file system parameters for ext2, ext3, and ext4 file systems. In addition, the following
tools are also useful in debugging and analyzing ext4 file systems:

d eb u g f s

D ebugs ext2, ext3, or ext4 file systems.

e2imag e

Saves critical ext2, ext3, or ext4 file system metadata to a file.

For more information about these utilities, refer to their respective man pages.

40
 Chapt er 6 . Bt rfs (T echnology Preview)

Chapter 6. Btrfs (Technology Preview)

Btrfs is a next generation Linux file system that offers advanced management, reliability, and
scalability features. It is unique in offering snapshots, compression, and integrated device
management.

Important

BTRFS is a Technology Preview in Red Hat Enterprise Linux 7.

6.1. Creat ing a bt rfs File Syst em

In order to make a basic btrfs file system, use the following command:

# mkfs. btrfs /dev/device

For more information on creating btrfs file systems with added devices and specifying multi-device
profiles for metadata and data, refer to Section 6.4, Integrated Volume Management of Multiple
D evices .

6.2. Mount ing a bt rfs file syst em

To mount any device in the btrfs file system use the following command:

# mount /dev/device /mount-point

Other useful mount options include:

d evice= /dev/name

Appending this option to the mount command tells btrfs to scan the named device for a btrfs
volume. This is used to ensure the mount will succeed as attempting to mount devices that
are not btrfs will cause the mount to fail.

Note

This does not mean all devices will be added to the file system, it only scans them.

max_in lin e= number

Use this option to set the maximum amount of space (in bytes) that can be used to inline
data within a metadata B-tree leaf. The default is 8192 bytes. For 4k pages it is limited to
3900 bytes due to additional headers that need to fit into the leaf.

allo c_st art = number

Use this option to set where in the disk allocations start.

t h read _p o o l= number

Use this option to assign the number of worker threads allocated.

41
St orage Administ rat ion G uide

Use this option to assign the number of worker threads allocated.

d iscard

Use this option to enable discard/TRIM on freed blocks.

n o acl

Use this option to disable the use of ACL's.

sp ace_cach e

Use this option to store the free space data on disk to make caching a block group faster.
This is a persistent change and is safe to boot into old kernels.

n o sp ace_cach e

Use this option to disable the above space_cache.

clear_cach e

Use this option to clear all the free space caches during mount. This is a safe option but will
trigger the space cache to be rebuilt. As such, leave the file system mounted in order to let
the rebuild process finish. This mount option is intended to be used once and only after
problems are apparent with the free space.

en o sp c_d eb u g

This option is used to debug problems with " no space left" .

reco very

Use this option to enable autorecovery upon mount.

6.3. Resiz ing a bt rfs file syst em

It is not possible to resize a btrfs file system but it is possible to resize each of the devices it uses. If
there is only one device in use then this works the same as resizing the file system. If there are
multiple devices in use then they must be manually resized to achieve the desired result.

Note

The unit size is not case specific; it accepts both G or g for GiB.

The command does not accept t for terabytes or p for petabytes. It only accepts k, m, and g .

Enlarging a bt rfs File Syst em

To enlarge the file system on a single device, use the command:

# btrfs filesystem resize amount /mount-point

For example:

42
 Chapt er 6 . Bt rfs (T echnology Preview)

# btrfs filesystem resize +200M /btrfssingle

Resize '/btrfssingle' of '+200M'

To enlarge a multi-device file system, the device to be enlarged must be specified. First, show all
devices that have a btrfs file system at a specified mount point:

# btrfs filesystem show /mount-point

For example:

# btrfs filesystem show /btrfstest

Label: none uuid: 755b41b7-7a20-4a24-abb3-45fdbed1ab39
Total devices 4 FS bytes used 192.00KiB
devid 1 size 1.00GiB used 224.75MiB path /dev/vdc
devid 2 size 524.00MiB used 204.75MiB path /dev/vdd
devid 3 size 1.00GiB used 8.00MiB path /dev/vde
devid 4 size 1.00GiB used 8.00MiB path /dev/vdf

Btrfs v3.16.2

Then, after identifying the d evi d of the device to be enlarged, use the following command:

# btrfs filesystem resize devid:amount /mount-point

For example:

# btrfs filesystem resize 2:+200M /btrfstest

Resize '/btrfstest/' of '2:+200M'

Note

The amount can also be max instead of a specified amount. This will use all remaining free
space on the device.

Shrinking a bt rfs File Syst em

To shrink the file system on a single device, use the command:

# btrfs filesystem resize amount /mount-point

For example:

# btrfs filesystem resize -200M /btrfssingle

Resize '/btrfssingle' of '-200M'

To shrink a multi-device file system, the device to be shrunk must be specified. First, show all devices
that have a btrfs file system at a specified mount point:

# btrfs filesystem show /mount-point

43
St orage Administ rat ion G uide

For example:

# btrfs filesystem show /btrfstest

Label: none uuid: 755b41b7-7a20-4a24-abb3-45fdbed1ab39
Total devices 4 FS bytes used 192.00KiB
devid 1 size 1.00GiB used 224.75MiB path /dev/vdc
devid 2 size 524.00MiB used 204.75MiB path /dev/vdd
devid 3 size 1.00GiB used 8.00MiB path /dev/vde
devid 4 size 1.00GiB used 8.00MiB path /dev/vdf

Btrfs v3.16.2

Then, after identifying the d evi d of the device to be shrunk, use the following command:

# btrfs filesystem resize devid:amount /mount-point

For example:

# btrfs filesystem resize 2:-200M /btrfstest

Resize '/btrfstest' of '2:-200M'

Set t he File Syst em Siz e

To set the file system to a specific size on a single device, use the command:

# btrfs filesystem resize amount /mount-point

For example:

# btrfs filesystem resize 700M /btrfssingle

Resize '/btrfssingle' of '700M'

To set the file system size of a multi-device file system, the device to be changed must be specified.
First, show all devices that have a btrfs file system at the specified mount point:

# btrfs filesystem show /mount-point

For example:

# btrfs filesystem show /btrfstest

Label: none uuid: 755b41b7-7a20-4a24-abb3-45fdbed1ab39
Total devices 4 FS bytes used 192.00KiB
devid 1 size 1.00GiB used 224.75MiB path /dev/vdc
devid 2 size 724.00MiB used 204.75MiB path /dev/vdd
devid 3 size 1.00GiB used 8.00MiB path /dev/vde
devid 4 size 1.00GiB used 8.00MiB path /dev/vdf

Btrfs v3.16.2

Then, after identifying the d evi d of the device to be changed, use the following command:

# btrfs filesystem resize devid:amount /mount-point

44
 Chapt er 6 . Bt rfs (T echnology Preview)

For example:

# btrfs filesystem resize 2:300M /btrfstest

Resize '/btrfstest' of '2:300M'

6.4 . Int egrat ed Volume Management of Mult iple Devices

A btrfs file system can be created on top of many devices, and more devices can be added after the
file system has been created. By default, metadata will be mirrored across two devices and data will
be striped across all devices present, however if only one device is present, metadata will be
duplicated on that device.

6.4 .1. File Syst em Creat ion wit h Mult iple Devices

The mkfs. btrfs command, as detailed in Section 6.1, Creating a btrfs File System , accepts the
options -d for data, and -m for metadata. Valid specifications are:

rai d 0

rai d 1

rai d 10

d up

si ng l e

The -m si ng l e option instructs that no duplication of metadata is done. This may be desired when
using hardware raid.

Note

Raid10 requires at least four devices to run correctly.

Examp le 6 .1. C reat in g a raid 10 b t rf s f ile syst em

Create a file system across four devices (metadata mirrored, data striped).

# mkfs.btrfs /dev/device1 /dev/device2 /dev/device3 /dev/device4

Stripe the metadata without mirroring.

# mkfs.btrfs -m raid0 /dev/device1 /dev/device2

Use raid10 for both data and metadata.

# mkfs.btrfs -m raid10 -d raid10 /dev/device1 /dev/device2 /dev/device3

/dev/device4

D o not duplicate metadata on a single drive.

45
St orage Administ rat ion G uide

# mkfs.btrfs -m single /dev/device

Use the si ng l e option to use the full capacity of each drive when the drives are different sizes.

# mkfs.btrfs -d single /dev/device1 /dev/device2 /dev/device3

To add a new device to an already created multi-device file system, use the following command:

# btrfs device add /dev/device1 /mount-point

After rebooting or reloading the btrfs module, use the btrfs d evi ce scan command to discover all
multi-device file systems. See Section 6.4.2, D evice scanning btrfs multiple devices for more
information.

6.4 .2. Device scanning bt rfs mult iple devices

Use btrfs d evi ce scan to scan all block devices under /d ev and probe for btrfs volumes. This
must be performed after loading the btrfs module if running with more than one device in a file system.

To scan all devices, use the following command:

# btrfs device scan

To scan a single device, use the following command:

# btrfs device scan /dev/device

6.4 .3. Adding new devices t o a bt rfs file syst em

Use the btrfs fi l esystem sho w command to list all the btrfs file systems and which devices they
include.

The btrfs d evi ce ad d command is used to add new devices to a mounted file system.

The btrfs fi l esystem bal ance command balances (restripes) the allocated extents across all
existing devices.

An example of all these commands together to add a new device is as follows:

Examp le 6 .2. Ad d a n ew d evice t o an b t rf s f ile syst em

First, create and mount a btrfs file system. Refer to Section 6.1, Creating a btrfs File System for
more information on how to create a btrfs file system, and to Section 6.2, Mounting a btrfs file
system for more information on how to mount a btrfs file system.

# mkfs.btrfs /dev/device1
# mount /dev/device1

Next, add a second device to the mounted btrfs file system.

# btrfs device add /dev/device2 /mount-point

46
 Chapt er 6 . Bt rfs (T echnology Preview)

The metadata and data on these devices are still stored only on /d ev/device1. It must now be
balanced to spread across all devices.

# btrfs filesystem balance /mount-point

Balancing a file system will take some time as it reads all of the file system's data and metadata and
rewrites it across the new device.

6.4 .4 . Convert ing a bt rfs file syst em

To convert a non-raid file system to a raid, add a device and run a balance filter that changes the
chunk allocation profile.

Examp le 6 .3. C o n vert in g a b t rf s f ile syst em

To convert an existing single device system, /d ev/sd b1 in this case, into a two device, raid1
system in order to protect against a single disk failure, use the following commands:

# mount /dev/sdb1 /mnt

# btrfs device add /dev/sdc1 /mnt
# btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt

Important

If the metadata is not converted from the single-device default, it remains as D UP. This does
not guarantee that copies of the block are on separate devices. If data is not converted it does
not have any redundant copies at all.

6.4 .5. Removing bt rfs devices

Use the btrfs d evi ce d el ete command to remove an online device. It redistributes any extents in
use to other devices in the file system in order to be safely removed.

Examp le 6 .4 . R emo vin g a d evice o n a b t rf s f ile syst em

First create and mount a few btrfs file systems.

# mkfs.btrfs /dev/sdb /dev/sdc /dev/sdd /dev/sde

# mount /dev/sdb /mnt

Add some data to the file system.

Finally, remove the required device.

# btrfs device delete /dev/sdc /mnt

6.4 .6. Replacing failed devices on a bt rfs file syst em

47
St orage Administ rat ion G uide

Section 6.4.5, Removing btrfs devices can be used to remove a failed device provided the super
block can still be read. However, if a device is missing or the super block corrupted, the file system
will need to be mounted in a degraded mode:

# mkfs.btrfs -m raid1 /dev/sdb /dev/sdc /dev/sdd /dev/sde

ssd is destroyed or removed, use -o degraded to force the mount

to ignore missing devices

# mount -o degraded /dev/sdb /mnt

'missing' is a special device name

# btrfs device delete missing /mnt

The command btrfs d evi ce d el ete mi ssi ng removes the first device that is described by the
file system metadata but not present when the file system was mounted.

Important

It is impossible to go below the minimum number of devices required for the specific raid
layout, even including the missing one. It may be required to add a new device in order to
remove the failed one.

For example, for a raid1 layout with two devices, if a device fails it is required to:

1. mount in degraded mode,

2. add a new device,
3. and, remove the missing device.

6.4 .7. Regist ering a bt rfs file syst em in /etc/fstab

If you do not have an i ni trd or it does not perform a btrfs device scan, it is possible to mount a
multi-volume btrfs file system by passing all the devices in the file system explicitly to the mo unt
command.

Examp le 6 .5. Examp le /etc/fstab en t ry

An example of a suitable /etc/fstab entry would be:

/dev/sdb /mnt btrfs

device=/dev/sdb,device=/dev/sdc,device=/dev/sdd,device=/dev/sde 0

Note that using universally unique identifiers (UUID s) also works and is more stable than using
device paths.

48
 Chapt er 6 . Bt rfs (T echnology Preview)

6.5. SSD Opt imiz at ion

Using the btrfs file system can optimize SSD . There are two ways this can be done.

The first way is mkfs. btrfs turns off metadata duplication on a single device when
/sys/bl o ck/device/q ueue/ro tati o nal is zero for the single specified device. This is
equivalent to specifying -m si ng l e on the command line. It can be overridden and duplicate
metadata forced by providing the -m d up option. D uplication is not required due to SSD firmware
potentially losing both copies. This wastes space and is a performance cost.

The second way is through a group of SSD mount options: ssd , no ssd , and ssd _spread .

The ssd option does several things:

It allows larger metadata cluster allocation.

It allocates data more sequentially where possible.

It disables btree leaf rewriting to match key and block order.

It commits log fragments without batching multiple processes.

Note

The ssd mount option only enables the ssd option. Use the no ssd option to disable it.

Some SSD s perform best when reusing block numbers often, while others perform much better when
clustering strictly allocates big chunks of unused space. By default, mo unt -o ssd will find
groupings of blocks where there are several free blocks that might have allocated blocks mixed in.
The command mo unt -o ssd _spread ensures there are no allocated blocks mixed in. This
improves performance on lower end SSD s.

Note

The ssd _spread option enables both the ssd and the ssd _spread options. Use the no ssd
to disable both these options.

The ssd _spread option is never automatically set if none of the ssd options are provided
and any of the devices are non-rotational.

These options will all need to be tested with your specific build to see if their use improves or reduces
performance, as each combination of SSD firmware and application loads are different.

6.6. bt rfs references

The man page btrfs(8) covers all important management commands. In particular this includes:

All the subvolume commands for managing snapshots.

The d evi ce commands for managing devices.

49
St orage Administ rat ion G uide

The scrub, bal ance, and d efrag ment commands.

The man page mkfs. btrfs(8) contains information on creating a btrfs file system including all the
options regarding it.

The man page btrfsck(8) for information regarding fsck on btrfs systems.

50
 Chapt er 7 . G lobal File Syst em 2

Chapter 7. Global File System 2

The Red Hat Global File System 2 (GFS2) is a native file system that interfaces directly with the Linux
kernel file system interface (VFS layer). When implemented as a cluster file system, GFS2 employs
distributed metadata and multiple journals.

GFS2 is based on 64-bit architecture, which can theoretically accommodate an 8 exabyte file system.
However, the current supported maximum size of a GFS2 file system is 100 TB. If a system requires
GFS2 file systems larger than 100 TB, contact your Red Hat service representative.

When determining the size of a file system, consider its recovery needs. Running the fsck command
on a very large file system can take a long time and consume a large amount of memory.
Additionally, in the event of a disk or disk-subsystem failure, recovery time is limited by the speed of
backup media.

When configured in a Red Hat Cluster Suite, Red Hat GFS2 nodes can be configured and managed
with Red Hat Cluster Suite configuration and management tools. Red Hat GFS2 then provides data
sharing among GFS2 nodes in a Red Hat cluster, with a single, consistent view of the file system
name space across the GFS2 nodes. This allows processes on different nodes to share GFS2 files in
the same way that processes on the same node can share files on a local file system, with no
discernible difference. For information about the Red Hat Cluster Suite, refer to Red Hat's Cluster
Administration guide.

A GFS2 must be built on a logical volume (created with LVM) that is a linear or mirrored volume.
Logical volumes created with LVM in a Red Hat Cluster suite are managed with CLVM (a cluster-wide
implementation of LVM), enabled by the CLVM daemon cl vmd , and running in a Red Hat Cluster
Suite cluster. The daemon makes it possible to use LVM2 to manage logical volumes across a
cluster, allowing all nodes in the cluster to share the logical volumes. For information on the Logical
Volume Manager, see Red Hat's Logical Volume Manager Administration guide.

The g fs2. ko kernel module implements the GFS2 file system and is loaded on GFS2 cluster nodes.

For comprehensive information on the creation and configuration of GFS2 file systems in clustered
and non-clustered storage, refer to Red Hat's Global File System 2 guide.

51
St orage Administ rat ion G uide

Chapter 8. Network File System (NFS)

A Network File System (NFS) allows remote hosts to mount file systems over a network and interact
with those file systems as though they are mounted locally. This enables system administrators to
consolidate resources onto centralized servers on the network.

This chapter focuses on fundamental NFS concepts and supplemental information.

8.1. How NFS Works

Currently, there are two major versions of NFS included in Red Hat Enterprise Linux. NFS version 3
(NFSv3) supports safe asynchronous writes and is more robust at error handling than the previous
NFSv2; it also supports 64-bit file sizes and offsets, allowing clients to access more than 2 GB of file
data. NFSv4 works through firewalls and on the Internet, no longer requires an rpcbi nd service,
supports ACLs, and utilizes stateful operations.

Red Hat Enterprise Linux 7 adds support for NFS version 4.1 (NFSv4.1), which provides a number of
performance and security enhancements, including client-side support for Parallel NFS (pNFS).
NFSv4.1 no longer requires a separate TCP connection for callbacks, which allows an NFS server to
grant delegations even when it cannot contact the client (for example, when NAT or a firewall
interferes). Additionally, NFSv4.1 now provides true exactly-once semantics (except for reboot
operations), preventing a previous issue whereby certain operations could return an inaccurate
result if a reply was lost and the operation was sent twice.

Red Hat Enterprise Linux 7 supports NFSv3, NFSv4.0, and NVSv4.1 clients. NFS clients attempt to
mount using NFSv4.0 by default, and fall back to NFSv3 if the mount operation is not successful.

Note

NFS version 2 (NFSv2) is no longer supported by Red Hat.

All versions of NFS can use Transmission Control Protocol (TCP) running over an IP network, with
NFSv4 requiring it. NFSv3 can use the User Datagram Protocol (UD P) running over an IP network to
provide a stateless network connection between the client and server.

When using NFSv3 with UD P, the stateless UD P connection (under normal conditions) has less
protocol overhead than TCP. This can translate into better performance on very clean, non-
congested networks. However, because UD P is stateless, if the server goes down unexpectedly, UD P
clients continue to saturate the network with requests for the server. In addition, when a frame is lost
with UD P, the entire RPC request must be retransmitted; with TCP, only the lost frame needs to be
resent. For these reasons, TCP is the preferred protocol when connecting to an NFS server.

The mounting and locking protocols have been incorporated into the NFSv4 protocol. The server
also listens on the well-known TCP port 2049. As such, NFSv4 does not need to interact with
rpcbi nd [1] , l o ckd , and rpc. statd daemons. The rpc. mo untd daemon is still required on the
NFS server to set up the exports, but is not involved in any over-the-wire operations.

52
 Chapt er 8 . Net work File Syst em (NFS)

Note

TCP is the default transport protocol for NFS version 2 and 3 under Red Hat Enterprise Linux.
UD P can be used for compatibility purposes as needed, but is not recommended for wide
usage. NFSv4 requires TCP.

All the RPC/NFS daemons have a ' -p' command line option that can set the port, making
firewall configuration easier.

After TCP wrappers grant access to the client, the NFS server refers to the /etc/expo rts
configuration file to determine whether the client is allowed to access any exported file systems. Once
verified, all file and directory operations are available to the user.

Important

In order for NFS to work with a default installation of Red Hat Enterprise Linux with a firewall
enabled, configure IPTables with the default TCP port 2049. Without proper IPTables
configuration, NFS will not function properly.

The NFS initialization script and rpc. nfsd process now allow binding to any specified port
during system start up. However, this can be error-prone if the port is unavailable, or if it
conflicts with another daemon.

8.1.1. Required Services

Red Hat Enterprise Linux uses a combination of kernel-level support and daemon processes to
provide NFS file sharing. All NFS versions rely on Remote Procedure Calls (RPC) between clients and
servers. RPC services under Red Hat Enterprise Linux 7 are controlled by the rpcbi nd service. To
share or mount NFS file systems, the following services work together depending on which version of
NFS is implemented:

Note

The po rtmap service was used to map RPC program numbers to IP address port number
combinations in earlier versions of Red Hat Enterprise Linux. This service is now replaced by
rpcbi nd in Red Hat Enterprise Linux 7 to enable IPv6 support.

nfs

systemctl start nfs starts the NFS server and the appropriate RPC processes to
service requests for shared NFS file systems.

n f slo ck

systemctl start nfs-l o ck activates a mandatory service that starts the appropriate
RPC processes allowing NFS clients to lock files on the server.

rp cb in d

53
St orage Administ rat ion G uide

rpcbi nd accepts port reservations from local RPC services. These ports are then made
available (or advertised) so the corresponding remote RPC services can access them.
rpcbi nd responds to requests for RPC services and sets up connections to the requested
RPC service. This is not used with NFSv4.

The following RPC processes facilitate NFS services:

rp c.mo u n t d

This process is used by an NFS server to process MO UNT requests from NFSv3 clients. It
checks that the requested NFS share is currently exported by the NFS server, and that the
client is allowed to access it. If the mount request is allowed, the rpc.mountd server replies
with a Success status and provides the Fi l e-Hand l e for this NFS share back to the NFS
client.

rp c.n f sd

rpc. nfsd allows explicit NFS versions and protocols the server advertises to be defined. It
works with the Linux kernel to meet the dynamic demands of NFS clients, such as providing
server threads each time an NFS client connects. This process corresponds to the nfs
service.

lo ckd

l o ckd is a kernel thread which runs on both clients and servers. It implements the Network
Lock Manager (NLM) protocol, which allows NFSv3 clients to lock files on the server. It is
started automatically whenever the NFS server is run and whenever an NFS file system is
mounted.

rp c.st at d

This process implements the Network Status Monitor (NSM) RPC protocol, which notifies NFS
clients when an NFS server is restarted without being gracefully brought down. rpc. statd
is started automatically by the nfsl o ck service, and does not require user configuration.
This is not used with NFSv4.

rp c.rq u o t ad

This process provides user quota information for remote users. rpc. rq uo tad is started
automatically by the nfs service and does not require user configuration.

rp c.id map d

rpc. i d mapd provides NFSv4 client and server upcalls, which map between on-the-wire
NFSv4 names (strings in the form of user@ domain) and local UID s and GID s. For i d mapd
to function with NFSv4, the /etc/i d mapd . co nf file must be configured. At a minimum, the
" D omain" parameter should be specified, which defines the NFSv4 mapping domain. If the
NFSv4 mapping domain is the same as the D NS domain name, this parameter can be
skipped. The client and server must agree on the NFSv4 mapping domain for ID mapping to
function properly.

54
 Chapt er 8 . Net work File Syst em (NFS)

Note

In Red Hat Enterprise Linux 7, only the NFSv4 server uses rpc. i d mapd . The NFSv4
client uses the keyring-based idmapper nfsi d map. nfsi d map is a stand-alone
program that is called by the kernel on-demand to perform ID mapping; it is not a
daemon. If there is a problem with nfsi d map does the client fall back to using
rpc. i d mapd . More information regarding nfsi d map can be found on the nfsidmap
man page.

8.2. pNFS

Support for Parallel NFS (pNFS) as part of the NFS v4.1 standard is available as of Red Hat
Enterprise Linux 6.4. The pNFS architecture improves the scalability of NFS, with possible
improvements to performance. That is, when a server implements pNFS as well, a client is able to
access data through multiple servers concurrently. It supports three storage protocols or layouts:
files, objects, and blocks.

Note

The protocol allows for three possible pNFS layout types: files, objects, and blocks. While the
Red Hat Enterprise Linux 6.4 client only supported the files layout type, Red Hat
Enterprise Linux 7 supports the files layout type, with objects and blocks layout types being
included as a technology preview.

To enable this functionality, use the following mount option on mounts from a pNFS-enabled server:

-o v4 . 1

After the server is pNFS-enabled, the nfs_l ayo ut_nfsv4 1_fi l es kernel is automatically loaded
on the first mount. The mount entry in the output should contain mi no rversi o n= 1. Use the
following command to verify the module was loaded:

$ l smo d | g rep nfs_l ayo ut_nfsv4 1_fi l es

For more information on pNFS, refer to: http://www.pnfs.com.

8.3. NFS Client Configurat ion

The mo unt command mounts NFS shares on the client side. Its format is as follows:

# mo unt -t nfs -o options server: /remote/export /local/directory

This command uses the following variables:

options

A comma-delimited list of mount options; refer to Section 8.5, Common NFS Mount
Options for details on valid NFS mount options.

55
St orage Administ rat ion G uide

server

The hostname, IP address, or fully qualified domain name of the server exporting the file
system you wish to mount

/remote/export

The file system or directory being exported from the server, that is, the directory you wish to
mount

/local/directory

The client location where /remote/export is mounted

The NFS protocol version used in Red Hat Enterprise Linux 7 is identified by the mo unt options
nfsvers or vers. By default, mo unt will use NFSv4 with mo unt -t nfs. If the server does not
support NFSv4, the client will automatically step down to a version supported by the server. If the
nfsvers/vers option is used to pass a particular version not supported by the server, the mount will
fail. The file system type nfs4 is also available for legacy reasons; this is equivalent to running mo unt
-t nfs -o nfsvers= 4 host: /remote/export /local/directory.

Refer to man mo unt for more details.

If an NFS share was mounted manually, the share will not be automatically mounted upon reboot.
Red Hat Enterprise Linux offers two methods for mounting remote file systems automatically at boot
time: the /etc/fstab file and the auto fs service. Refer to Section 8.3.1, Mounting NFS File
Systems using /etc/fstab and Section 8.4, auto fs for more information.

8.3.1. Mount ing NFS File Syst ems using /etc/fstab

An alternate way to mount an NFS share from another machine is to add a line to the /etc/fstab
file. The line must state the hostname of the NFS server, the directory on the server being exported,
and the directory on the local machine where the NFS share is to be mounted. You must be root to
modify the /etc/fstab file.

Examp le 8.1. Syn t ax examp le

The general syntax for the line in /etc/fstab is as follows:

server:/usr/local/pub /pub nfs defaults 0 0

The mount point /pub must exist on the client machine before this command can be executed. After
adding this line to /etc/fstab on the client system, use the command mo unt /pub, and the mount
point /pub is mounted from the server.

A valid /etc/fstab entry to mount an NFS export should contain the following information:

server:/remote/export /local/directory nfs options 0 0

The variables server, /remote/export, /local/directory, and options are the same ones used when
manually mounting an NFS share. Refer to Section 8.3, NFS Client Configuration for a definition of
each variable.

56
 Chapt er 8 . Net work File Syst em (NFS)

Note

The mount point /local/directory must exist on the client before /etc/fstab is read. Otherwise,
the mount will fail.

For more information about /etc/fstab, refer to man fstab.

8.4 . auto fs

One drawback to using /etc/fstab is that, regardless of how infrequently a user accesses the NFS
mounted file system, the system must dedicate resources to keep the mounted file system in place.
This is not a problem with one or two mounts, but when the system is maintaining mounts to many
systems at one time, overall system performance can be affected. An alternative to /etc/fstab is to
use the kernel-based auto mo unt utility. An automounter consists of two components:

a kernel module that implements a file system, and

a user-space daemon that performs all of the other functions.

The auto mo unt utility can mount and unmount NFS file systems automatically (on-demand
mounting), therefore saving system resources. It can be used to mount other file systems including
AFS, SMBFS, CIFS, and local file systems.

Important

The nfs-utils package is now a part of both the 'NFS file server' and the 'Network File System
Client' groups. As such, it is no longer installed by default with the Base group. Ensure that
nfs-utils is installed on the system first before attempting to automount an NFS share.

autofs is also part of the 'Network File System Client' group.

auto fs uses /etc/auto . master (master map) as its default primary configuration file. This can be
changed to use another supported network source and name using the auto fs configuration (in
/etc/sysco nfi g /auto fs) in conjunction with the Name Service Switch (NSS) mechanism. An
instance of the auto fs version 4 daemon was run for each mount point configured in the master
map and so it could be run manually from the command line for any given mount point. This is not
possible with auto fs version 5, because it uses a single daemon to manage all configured mount
points; as such, all automounts must be configured in the master map. This is in line with the usual
requirements of other industry standard automounters. Mount point, hostname, exported directory,
and options can all be specified in a set of files (or other supported network sources) rather than
configuring them manually for each host.

8.4 .1. Improvement s in aut ofs Version 5 over Version 4

auto fs version 5 features the following enhancements over version 4:

D irect map su p p o rt

57
St orage Administ rat ion G uide

D irect maps in auto fs provide a mechanism to automatically mount file systems at

arbitrary points in the file system hierarchy. A direct map is denoted by a mount point of /-
in the master map. Entries in a direct map contain an absolute path name as a key (instead
of the relative path names used in indirect maps).

Laz y mo u n t an d u n mo u n t su p p o rt

Multi-mount map entries describe a hierarchy of mount points under a single key. A good
example of this is the -ho sts map, commonly used for automounting all exports from a
host under /net/host as a multi-mount map entry. When using the -ho sts map, an l s of
/net/host will mount autofs trigger mounts for each export from host. These will then
mount and expire them as they are accessed. This can greatly reduce the number of active
mounts needed when accessing a server with a large number of exports.

En h an ced LD AP su p p o rt

The auto fs configuration file (/etc/sysco nfi g /auto fs) provides a mechanism to
specify the auto fs schema that a site implements, thus precluding the need to determine
this via trial and error in the application itself. In addition, authenticated binds to the LD AP
server are now supported, using most mechanisms supported by the common LD AP server
implementations. A new configuration file has been added for this support:
/etc/auto fs_l d ap_auth. co nf. The default configuration file is self-documenting, and
uses an XML format.

Pro p er u se o f t h e N ame Service Swit ch ( nsswi tch) co n f ig u rat io n .

The Name Service Switch configuration file exists to provide a means of determining from
where specific configuration data comes. The reason for this configuration is to allow
administrators the flexibility of using the back-end database of choice, while maintaining a
uniform software interface to access the data. While the version 4 automounter is becoming
increasingly better at handling the NSS configuration, it is still not complete. Autofs version
5, on the other hand, is a complete implementation.

Refer to man nsswi tch. co nf for more information on the supported syntax of this file. Not
all NSS databases are valid map sources and the parser will reject ones that are invalid.
Valid sources are files, yp, ni s, ni spl us, l d ap, and hesi o d .

Mu lt ip le mast er map en t ries p er au t o f s mo u n t p o in t

One thing that is frequently used but not yet mentioned is the handling of multiple master
map entries for the direct mount point /-. The map keys for each entry are merged and
behave as one map.

Examp le 8.2. Mu lt ip le mast er map en t ries p er au t o f s mo u n t p o in t

An example is seen in the connectathon test maps for the direct mounts below:

/- /tmp/auto_dcthon
/- /tmp/auto_test3_direct
/- /tmp/auto_test4_direct

8.4 .2. auto fs Configurat ion

The primary configuration file for the automounter is /etc/auto . master, also referred to as the

58
 Chapt er 8 . Net work File Syst em (NFS)

master map which may be changed as described in the Section 8.4.1, Improvements in autofs
Version 5 over Version 4 . The master map lists auto fs-controlled mount points on the system, and
their corresponding configuration files or network sources known as automount maps. The format of
the master map is as follows:

mount-point map-name options

The variables used in this format are:

mount-point

The auto fs mount point, /ho me, for example.

map-name

The name of a map source which contains a list of mount points, and the file system
location from which those mount points should be mounted. The syntax for a map entry is
described below.

options

If supplied, these will apply to all entries in the given map provided they don't themselves
have options specified. This behavior is different from auto fs version 4 where options were
cumulative. This has been changed to implement mixed environment compatibility.

Examp le 8.3. /etc/auto . master f ile

The following is a sample line from /etc/auto . master file (displayed with cat
/etc/auto . master):

/home /etc/auto.misc

The general format of maps is similar to the master map, however the " options" appear between
the mount point and the location instead of at the end of the entry as in the master map:

mount-point [options] location

The variables used in this format are:

mount-point

This refers to the auto fs mount point. This can be a single directory name for an indirect
mount or the full path of the mount point for direct mounts. Each direct and indirect map
entry key (mount-point above) may be followed by a space separated list of offset
directories (sub directory names each beginning with a " /" ) making them what is known as
a multi-mount entry.

options

Whenever supplied, these are the mount options for the map entries that do not specify their
own options.

location

59
St orage Administ rat ion G uide

This refers to the file system location such as a local file system path (preceded with the
Sun map format escape character " :" for map names beginning with " /" ), an NFS file system
or other valid file system location.

The following is a sample of contents from a map file (for example, /etc/auto . mi sc):

payroll -fstype=nfs personnel:/dev/hda3

sales -fstype=ext3 :/dev/hda4

The first column in a map file indicates the auto fs mount point (sal es and payro l l from the
server called perso nnel ). The second column indicates the options for the auto fs mount while the
third column indicates the source of the mount. Following the above configuration, the autofs mount
points will be /ho me/payro l l and /ho me/sal es. The -fstype= option is often omitted and is
generally not needed for correct operation.

The automounter will create the directories if they do not exist. If the directories exist before the
automounter was started, the automounter will not remove them when it exits. You can start or restart
the automount daemon by issuing either of the following two commands:

servi ce auto fs start (if the automount daemon has stopped)

servi ce auto fs restart

Using the above configuration, if a process requires access to an auto fs unmounted directory such
as /ho me/payro l l /20 0 6 /Jul y. sxc, the automount daemon automatically mounts the directory.
If a timeout is specified, the directory will automatically be unmounted if the directory is not accessed
for the timeout period.

You can view the status of the automount daemon by issuing the following command:

# servi ce auto fs status

8.4 .3. Overriding or Augment ing Sit e Configurat ion Files

It can be useful to override site defaults for a specific mount point on a client system. For example,
consider the following conditions:

Automounter maps are stored in NIS and the /etc/nsswi tch. co nf file has the following
directive:

automount: files nis

The auto . master file contains the following

+auto.master

The NIS auto . master map file contains the following:

/home auto.home

The NIS auto . ho me map contains the following:

beth fileserver.example.com:/export/home/beth
joe fileserver.example.com:/export/home/joe
* fileserver.example.com:/export/home/&

60
 Chapt er 8 . Net work File Syst em (NFS)

The file map /etc/auto . ho me does not exist.

Given these conditions, let's assume that the client system needs to override the NIS map
auto . ho me and mount home directories from a different server. In this case, the client will need to
use the following /etc/auto . master map:

/home / etc/auto.home
+auto.master

The /etc/auto . ho me map contains the entry:

* labserver.example.com:/export/home/&

Because the automounter only processes the first occurrence of a mount point, /ho me will contain
the contents of /etc/auto . ho me instead of the NIS auto . ho me map.

Alternatively, to augment the site-wide auto . ho me map with just a few entries, create an
/etc/auto . ho me file map, and in it put the new entries. At the end, include the NIS auto . ho me
map. Then the /etc/auto . ho me file map will look similar to:

mydir someserver:/export/mydir
+auto.home

Given the NIS auto . ho me map listed above, l s /ho me would now output:

beth joe mydir

This last example works as expected because auto fs does not include the contents of a file map of
the same name as the one it is reading. As such, auto fs moves on to the next map source in the
nsswi tch configuration.

8.4 .4 . Using LDAP t o St ore Aut omount er Maps

LD AP client libraries must be installed on all systems configured to retrieve automounter maps from
LD AP. On Red Hat Enterprise Linux, the o penl d ap package should be installed automatically as a
dependency of the auto mo unter. To configure LD AP access, modify
/etc/o penl d ap/l d ap. co nf. Ensure that BASE, URI, and schema are set appropriately for your
site.

The most recently established schema for storing automount maps in LD AP is described by
rfc230 7bi s. To use this schema it is necessary to set it in the auto fs configuration
(/etc/sysco nfi g /auto fs) by removing the comment characters from the schema definition. For
example:

Examp le 8.4 . Set t in g au t o f s co n f ig u rat io n

DEFAULT_MAP_OBJECT_CLASS="automountMap"
DEFAULT_ENTRY_OBJECT_CLASS="automount"
DEFAULT_MAP_ATTRIBUTE="automountMapName"
DEFAULT_ENTRY_ATTRIBUTE="automountKey"
DEFAULT_VALUE_ATTRIBUTE="automountInformation"

61
St orage Administ rat ion G uide

Ensure that these are the only schema entries not commented in the configuration. The
auto mo untKey replaces the cn attribute in the rfc230 7bi s schema. An LD IF of a sample
configuration is described below:

Examp le 8.5. LD F co n f ig u rat io n

# extended LDIF
#
# LDAPv3
# base <> with scope subtree
# filter: (& (objectclass=automountMap)(automountMapName=auto.master))
# requesting: ALL
#

# auto.master, example.com
dn: automountMapName=auto.master,dc=example,dc=com
objectClass: top
objectClass: automountMap
automountMapName: auto.master

# extended LDIF
#
# LDAPv3
# base <automountMapName=auto.master,dc=example,dc=com> with scope
subtree
# filter: (objectclass=automount)
# requesting: ALL
#

# /home, auto.master, example.com

dn: automountMapName=auto.master,dc=example,dc=com
objectClass: automount
cn: /home

automountKey: /home
automountInformation: auto.home

# extended LDIF
#
# LDAPv3
# base <> with scope subtree
# filter: (& (objectclass=automountMap)(automountMapName=auto.home))
# requesting: ALL
#

# auto.home, example.com
dn: automountMapName=auto.home,dc=example,dc=com
objectClass: automountMap
automountMapName: auto.home

# extended LDIF
#
# LDAPv3
# base <automountMapName=auto.home,dc=example,dc=com> with scope
subtree

62
 Chapt er 8 . Net work File Syst em (NFS)

# filter: (objectclass=automount)
# requesting: ALL
#

# foo, auto.home, example.com

dn: automountKey=foo,automountMapName=auto.home,dc=example,dc=com
objectClass: automount
automountKey: foo
automountInformation: filer.example.com:/export/foo

# /, auto.home, example.com
dn: automountKey=/,automountMapName=auto.home,dc=example,dc=com
objectClass: automount
automountKey: /
automountInformation: filer.example.com:/export/&

8.5. Common NFS Mount Opt ions

Beyond mounting a file system with NFS on a remote host, it is also possible to specify other options
at mount time to make the mounted share easier to use. These options can be used with manual
mo unt commands, /etc/fstab settings, and auto fs.

The following are options commonly used for NFS mounts:

in t r

Allows NFS requests to be interrupted if the server goes down or cannot be reached.

lo o ku p cach e= mode

Specifies how the kernel should manage its cache of directory entries for a given mount
point. Valid arguments for mode are al l , no ne, or po s/po si ti ve.

n f svers= version

Specifies which version of the NFS protocol to use, where version is 2, 3, or 4. This is useful
for hosts that run multiple NFS servers. If no version is specified, NFS uses the highest
version supported by the kernel and mo unt command.

The option vers is identical to nfsvers, and is included in this release for compatibility
reasons.

n o acl

Turns off all ACL processing. This may be needed when interfacing with older versions of
Red Hat Enterprise Linux, Red Hat Linux, or Solaris, since the most recent ACL technology
is not compatible with older systems.

n o lo ck

D isables file locking. This setting is occasionally required when connecting to older NFS
servers.

n o exec

Prevents execution of binaries on mounted file systems. This is useful if the system is
mounting a non-Linux file system containing incompatible binaries.

63
St orage Administ rat ion G uide

n o su id

D isables set-user-i d enti fi er or set-g ro up-i d enti fi er bits. This prevents remote
users from gaining higher privileges by running a setui d program.

p o rt = num

po rt= num Specifies the numeric value of the NFS server port. If num is 0 (the default),
then mo unt queries the remote host's rpcbi nd service for the port number to use. If the
remote host's NFS daemon is not registered with its rpcbi nd service, the standard NFS
port number of TCP 2049 is used instead.

rsiz e= num an d wsiz e= num

These settings speed up NFS communication for reads (rsi ze) and writes (wsi ze) by
setting a larger data block size (num, in bytes), to be transferred at one time. Be careful
when changing these values; some older Linux kernels and network cards do not work well
with larger block sizes. For NFSv3, the default values for both parameters is set to 8192.
For NFSv4, the default values for both parameters is set to 32768.

sec= mode

Its default setting is sec= sys, which uses local UNIX UID s and GID s. These use AUT H_SY S
to authenticate NFS operations."

sec= krb5 uses Kerberos V5 instead of local UNIX UID s and GID s to authenticate users.

sec= krb5i uses Kerberos V5 for user authentication and performs integrity checking of
NFS operations using secure checksums to prevent data tampering.

sec= krb5p uses Kerberos V5 for user authentication, integrity checking, and encrypts NFS
traffic to prevent traffic sniffing. This is the most secure setting, but it also involves the most
performance overhead.

t cp

Instructs the NFS mount to use the TCP protocol.

udp

Instructs the NFS mount to use the UD P protocol.

For a complete list of options and more detailed information on each one, refer to man mo unt and
man nfs.

8.6. St art ing and St opping NFS

To run an NFS server that is not configured to use only NFSv4, the rpcbi nd [1] service must be
running. To verify that rpcbi nd is active, use the following command:

# systemctl status rpcbi nd

If the rpcbi nd service is running, then the nfs service can be started. To start an NFS server, use
the following command:

# systemctl start nfs

64
 Chapt er 8 . Net work File Syst em (NFS)

To enable NFS to start at boot, use the following command:

# systemctl enabl e nfs-server

Note

For NFSv3, if NFS is set to start at boot time, the nfs-l o ck service needs to be enabled. On
Red Hat Enterprise Linux 7.1 and later, nfs-l o ck starts automatically if needed, and an
attempt to enable it manually fails. On Red Hat Enterprise Linux 7.0, check the status by
running systemctl status nfs-l o ck. If nfs-l o ck is not enabled, run systemctl
start nfs-l o ck. To set nfs-l o ck to automatically start on boot on Red Hat
Enterprise Linux 7.0, run systemctl enabl e nfs-l o ck.

To stop the server, use:

# systemctl sto p nfs

The restart option is a shorthand way of stopping and then starting NFS. This is the most efficient
way to make configuration changes take effect after editing the configuration file for NFS. To restart
the server type:

# systemctl restart nfs

After you edit the /etc/sysco nfi g /nfs file, restart the nfs-config service by running the following
command for the new values to take effect:

# systemctl restart nfs-co nfi g

The try-restart command only starts nfs if it is currently running. This command is the equivalent
of co nd restart (conditional restart) in Red Hat init scripts and is useful because it does not start the
daemon if NFS is not running.

To conditionally restart the server type:

# systemctl try-restart nfs

To reload the NFS server configuration file without restarting the service type:

# systemctl rel o ad nfs

8.7. NFS Server Configurat ion

There are two ways to configure exports on an NFS server:

Manually editing the NFS configuration file, that is, /etc/expo rts, and

through the command line, that is, by using the command expo rtfs

8.7.1. T he /etc/expo rts Configurat ion File

65
St orage Administ rat ion G uide

The /etc/expo rts file controls which file systems are exported to remote hosts and specifies
options. It follows the following syntax rules:

Blank lines are ignored.

To add a comment, start a line with the hash mark (#).

You can wrap long lines with a backslash (\).

Each exported file system should be on its own individual line.

Any lists of authorized hosts placed after an exported file system must be separated by space
characters.

Options for each of the hosts must be placed in parentheses directly after the host identifier,
without any spaces separating the host and the first parenthesis.

Each entry for an exported file system has the following structure:

export host(options)

The aforementioned structure uses the following variables:

export

The directory being exported

host

The host or network to which the export is being shared

options

The options to be used for host

It is possible to specify multiple hosts, along with specific options for each host. To do so, list them
on the same line as a space-delimited list, with each hostname followed by its respective options (in
parentheses), as in:

export host1(options1) host2(options2) host3(options3)

For information on different methods for specifying hostnames, refer to Section 8.7.5, Hostname
Formats .

In its simplest form, the /etc/expo rts file only specifies the exported directory and the hosts
permitted to access it, as in the following example:

Examp le 8.6 . T h e /etc/expo rts f ile

/exported/directory bob.example.com

Here, bo b. exampl e. co m can mount /expo rted /d i recto ry/ from the NFS server. Because no
options are specified in this example, NFS will use default settings.

The default settings are:

ro

66
 Chapt er 8 . Net work File Syst em (NFS)

The exported file system is read-only. Remote hosts cannot change the data shared on the
file system. To allow hosts to make changes to the file system (that is, read/write), specify the
rw option.

syn c

The NFS server will not reply to requests before changes made by previous requests are
written to disk. To enable asynchronous writes instead, specify the option async.

wd elay

The NFS server will delay writing to the disk if it suspects another write request is imminent.
This can improve performance as it reduces the number of times the disk must be accesses
by separate write commands, thereby reducing write overhead. To disable this, specify the
no _wd el ay. no _wd el ay is only available if the default sync option is also specified.

ro o t _sq u ash

This prevents root users connected remotely (as opposed to locally) from having root
privileges; instead, the NFS server will assign them the user ID nfsno bo d y. This effectively
" squashes" the power of the remote root user to the lowest local user, preventing possible
unauthorized writes on the remote server. To disable root squashing, specify
no _ro o t_sq uash.

To squash every remote user (including root), use al l _sq uash. To specify the user and group ID s
that the NFS server should assign to remote users from a particular host, use the ano nui d and
ano ng i d options, respectively, as in:

export host(anonuid=uid,anongid=gid)

Here, uid and gid are user ID number and group ID number, respectively. The ano nui d and
ano ng i d options allow you to create a special user and group account for remote NFS users to
share.

By default, access control lists (ACLs) are supported by NFS under Red Hat Enterprise Linux. To
disable this feature, specify the no _acl option when exporting the file system.

Each default for every exported file system must be explicitly overridden. For example, if the rw option
is not specified, then the exported file system is shared as read-only. The following is a sample line
from /etc/expo rts which overrides two default options:

/ano ther/expo rted /d i recto ry 19 2. 16 8. 0 . 3(rw,async)

In this example 19 2. 16 8. 0 . 3 can mount /ano ther/expo rted /d i recto ry/ read/write and all
writes to disk are asynchronous. For more information on exporting options, refer to man expo rtfs.

Other options are available where no default value is specified. These include the ability to disable
sub-tree checking, allow access from insecure ports, and allow insecure file locks (necessary for
certain early NFS client implementations). Refer to man expo rts for details on these less-used
options.

67
St orage Administ rat ion G uide

Important

The format of the /etc/expo rts file is very precise, particularly in regards to use of the space
character. Remember to always separate exported file systems from hosts and hosts from one
another with a space character. However, there should be no other space characters in the file
except on comment lines.

For example, the following two lines do not mean the same thing:

/home bob.example.com(rw)
/home bob.example.com (rw)

The first line allows only users from bo b. exampl e. co m read/write access to the /ho me
directory. The second line allows users from bo b. exampl e. co m to mount the directory as
read-only (the default), while the rest of the world can mount it read/write.

8.7.2. T he expo rtfs Command

Every file system being exported to remote users with NFS, as well as the access level for those file
systems, are listed in the /etc/expo rts file. When the nfs service starts, the
/usr/sbi n/expo rtfs command launches and reads this file, passes control to rpc. mo untd (if
NFSv2 or NFSv3) for the actual mounting process, then to rpc. nfsd where the file systems are then
available to remote users.

When issued manually, the /usr/sbi n/expo rtfs command allows the root user to selectively
export or unexport directories without restarting the NFS service. When given the proper options, the
/usr/sbi n/expo rtfs command writes the exported file systems to /var/l i b/nfs/xtab. Since
rpc. mo untd refers to the xtab file when deciding access privileges to a file system, changes to the
list of exported file systems take effect immediately.

The following is a list of commonly-used options available for /usr/sbi n/expo rtfs:

-r

Causes all directories listed in /etc/expo rts to be exported by constructing a new export
list in /etc/l i b/nfs/xtab. This option effectively refreshes the export list with any
changes made to /etc/expo rts.

-a

Causes all directories to be exported or unexported, depending on what other options are
passed to /usr/sbi n/expo rtfs. If no other options are specified,
/usr/sbi n/expo rtfs exports all file systems specified in /etc/expo rts.

- o file-systems

Specifies directories to be exported that are not listed in /etc/expo rts. Replace file-
systems with additional file systems to be exported. These file systems must be formatted in
the same way they are specified in /etc/expo rts. This option is often used to test an
exported file system before adding it permanently to the list of file systems to be exported.
Refer to Section 8.7.1, The /etc/expo rts Configuration File for more information on
/etc/expo rts syntax.

-i

68
 Chapt er 8 . Net work File Syst em (NFS)

Ignores /etc/expo rts; only options given from the command line are used to define
exported file systems.

-u

Unexports all shared directories. The command /usr/sbi n/expo rtfs -ua suspends
NFS file sharing while keeping all NFS daemons up. To re-enable NFS sharing, use
expo rtfs -r.

-v

Verbose operation, where the file systems being exported or unexported are displayed in
greater detail when the expo rtfs command is executed.

If no options are passed to the expo rtfs command, it displays a list of currently exported file
systems. For more information about the expo rtfs command, refer to man expo rtfs.

8 .7 .2 .1 . Using expo rtfs wit h NFSv4

In Red Hat Enterprise Linux 7, no extra steps are required to configure NFSv4 exports as any
filesystems mentioned are automatically available to NFSv3 and NFSv4 clients using the same path.
This was not the case in previous versions.

To prevent clients from using NFSv4, turn it off by setting R P C NFSD AR G S= -N 4 in

/etc/sysco nfi g /nfs.

8.7.3. Running NFS Behind a Firewall

NFS requires rpcbi nd , which dynamically assigns ports for RPC services and can cause problems
for configuring firewall rules. To allow clients to access NFS shares behind a firewall, edit the
/etc/sysco nfi g /nfs file to set which ports the RPC services run on.

The /etc/sysco nfi g /nfs file does not exist by default on all systems. If /etc/sysco nfi g /nfs
does not exist, create it and specify the following:

R PC MO U N T D O PT S= "- p port"

This adds " -p port" to the rpc.mount command line: rpc. mo unt -p port.

To specify the ports to be used by the nl o ckmg r service, set the port number for the nlm_tcpport
and nlm_udpport options in the /etc/mo d pro be. d /l o ckd . co nf file.

If NFS fails to start, check /var/l o g /messag es. Commonly, NFS fails to start if you specify a port
number that is already in use. After editing /etc/sysco nfi g /nfs, you need to restart the nfs-
co nfi g service for the new values to take effect in Red Hat Enterprise Linux 7.2 and prior by
running:

# systemctl restart nfs-config

Then, restart the NFS server:

# systemctl restart nfs-server

Run rpci nfo -p to confirm the changes have taken effect.

69
St orage Administ rat ion G uide

Note

To allow NFSv4.0 callbacks to pass through firewalls set

/pro c/sys/fs/nfs/nfs_cal l back_tcppo rt and allow the server to connect to that port
on the client.

This process is not needed for NFSv4.1 or higher, and the other ports for mo untd , statd , and
l o ckd are not required in a pure NFSv4 environment.

8 .7 .3.1 . Disco ve ring NFS e xpo rt s

There are two ways to discover which file systems an NFS server exports.

First, on any server that supports NFSv2 or NFSv3, use the sho wmo unt command:

$ showmount -e myserver
Export list for mysever
/exports/foo
/exports/bar

Second, on any server that supports NFSv4, mount / and look around.

# mount myserver:/ /mnt/

#cd /mnt/
exports
# ls exports
foo
bar

On servers that support both NFSv4 and either NFSv2 or NFSv3, both methods will work and give the
same results.

Note

Before Red Hat Enterprise Linux 6 on older NFS servers, depending on how they are
configured, it is possible to export filesystems to NFSv4 clients at different paths. Because
these servers do not enable NFSv4 by default this should not normally be a problem.

8.7.4 . Accessing RPC quot a t hrough a Firewall

If you export a file system that uses disk quotas, you can use the quota Remote Procedure Call (RPC)
service to provide disk quota data to NFS clients.

Pro ced u re 8.1. Makin g R PC q u o t a Accessib le B eh in d a Firewall

1. To enable the rpc-rq uo tad service, enter:

# systemctl enabl e rpc-rq uo tad

70
 Chapt er 8 . Net work File Syst em (NFS)

2. To start the rpc-rq uo tad service, enter:

# systemctl start rpc-rq uo tad

Note that rpc-rq uo tad is, if enabled, started automatically after starting the nfs-server
service.

3. To make the quota RPC service accessible behind a firewall, UD P or TCP port 875 need to be
open. The default port number is defined in the /etc/servi ces file.

You can override the default port number by appending -p port-number to the
R P C R Q UO T AD O P T S variable in the /etc/sysco nfi g /rpc-rq uo tad file.

4. Restart rpc-rq uo tad for changes in the /etc/sysco nfi g /rpc-rq uo tad file to take effect:

# systemctl restart rpc-rq uo tad

Set t ing Quot as from Remot e Host s

By default, quotas can only be read by remote hosts. To allow setting quotas, append the -S option
to the R P C R Q UO T AD O P T S variable in the /etc/sysco nfi g /rpc-rq uo tad file.

Restart rpc-rq uo tad for changes in the /etc/sysco nfi g /rpc-rq uo tad file to take effect:

# systemctl restart rpc-rq uo tad

8.7.5. Host name Format s

The host(s) can be in the following forms:

Sin g le mach in e

A fully-qualified domain name (that can be resolved by the server), hostname (that can be
resolved by the server), or an IP address.

Series o f mach in es sp ecif ied wit h wild card s

Use the * or ? character to specify a string match. Wildcards are not to be used with IP
addresses; however, they may accidentally work if reverse D NS lookups fail. When
specifying wildcards in fully qualified domain names, dots (. ) are not included in the
wildcard. For example, *. exampl e. co m includes o ne. exampl e. co m but does not
i ncl ud e o ne. two . exampl e. co m.

IP n et wo rks

Use a.b.c.d/z, where a.b.c.d is the network and z is the number of bits in the netmask (for
example 192.168.0.0/24). Another acceptable format is a.b.c.d/netmask, where a.b.c.d is the
network and netmask is the netmask (for example, 192.168.100.8/255.255.255.0).

N et g ro u p s

Use the format @group-name, where group-name is the NIS netgroup name.

8.7.6. Enabling NFS over RDMA (NFSoRDMA)

71
St orage Administ rat ion G uide

The remote direct memory access (RD MA) service works automatically in Red Hat Enterprise Linux 7 if
there is RD MA-capable hardware present.

Install the rdma package. The /etc/rd ma/rd ma. co nf file contains a line that sets
XPRTRDMA_LOAD=yes by default, which requests the rd ma service to load the NFSoRD MA client
module.

To enable automatic loading of NFSoRD MA server modules, add SVCRDMA_LOAD=yes on a new line
in /etc/rd ma/rd ma. co nf.

RPCNFSDARGS="--rdma=20049" in the /etc/sysco nfi g /nfs file specifies the port number on
which the NFSoRD MA service listens for clients. RFC 5667 specifies that servers must listen on port
20 0 4 9 when providing NFSv4 services over RD MA.

Restart the nfs service after editing the /etc/rd ma/rd ma. co nf file:

# service nfs restart

Note that with earlier kernel versions, a system reboot is needed after editing
/etc/rd ma/rd ma. co nf for the changes to take effect.

8.8. Securing NFS

NFS is suitable for transparent sharing of entire file systems with a large number of known hosts.
However, with ease-of-use comes a variety of potential security problems. To minimize NFS security
risks and protect data on the server, consider the following sections when exporting NFS file systems
on a server or mounting them on a client.

8.8.1. NFS Securit y wit h AUT H_SYS and export cont rols

Traditionally, NFS has given two options in order to control access to exported files.

First, the server restricts which hosts are allowed to mount which filesystems either by IP address or
by host name.

Second, the server enforces file system permissions for users on NFS clients in the same way it does
local users. Traditionally it does this using AUT H_SY S (also called AUT H_UNIX) which relies on the
client to state the UID and GID 's of the user. Be aware that this means a malicious or misconfigured
client can easily get this wrong and allow a user access to files that it should not.

To limit the potential risks, administrators often allow read-only access or squash user permissions
to a common user and group ID . Unfortunately, these solutions prevent the NFS share from being
used in the way it was originally intended.

Additionally, if an attacker gains control of the D NS server used by the system exporting the NFS file
system, the system associated with a particular hostname or fully qualified domain name can be
pointed to an unauthorized machine. At this point, the unauthorized machine is the system permitted
to mount the NFS share, since no username or password information is exchanged to provide
additional security for the NFS mount.

Wildcards should be used sparingly when exporting directories through NFS, as it is possible for the
scope of the wildcard to encompass more systems than intended.

It is also possible to restrict access to the rpcbi nd [1] service with TCP wrappers. Creating rules with
i ptabl es can also limit access to ports used by rpcbi nd , rpc. mo untd , and rpc. nfsd .

72
 Chapt er 8 . Net work File Syst em (NFS)

For more information on securing NFS and rpcbi nd , refer to man i ptabl es.

8.8.2. NFS securit y wit h AUT H_G SS

NFSv4 revolutionized NFS security by mandating the implementation of RPCSEC_GSS and the
Kerberos version 5 GSS-API mechanism. However, RPCSEC_GSS and the Kerberos mechanism are
also available for all versions of NFS. In FIPS mode, only FIPS-approved algorithms can be used.

Unlike AUTH_SYS, with the RPCSEC_GSS Kerberos mechanism, the server does not depend on the
client to correctly represent which user is accessing the file. Instead, cryptography is used to
authenticate users to the server, which prevents a malicious client from impersonating a user without
having that user's Kerberos credentials. Using the RPCSEC_GSS Kerberos mechanism is the most
straightforward way to secure mounts because after configuring Kerberos, no additional setup is
needed.

Configuring Kerberos

Before configuring an NFSv4 Kerberos-aware server, you need to install and configure a Kerberos
Key D istribution Centre (KD C). Kerberos is a network authentication system that allows clients and
servers to authenticate to each other by using symmetric encryption and a trusted third party, the
KD C. Red Hat recommends using Identity Management (IdM) for setting up Kerberos.

Pro ced u re 8.2. C o n f ig u rin g an N FS Server an d C lien t f o r Id M t o U se R PC SEC _G SS

1. Create the nfs/ho stname. domain@REALM principal on the NFS server side.

Create the ho st/ho stname. domain@REALM principal on both the server and the client
side.

Add the corresponding keys to keytabs for the client and server.

For instructions, see the Adding and Editing Service Entries and Keytabs and Setting up a
Kerberos-aware NFS Server sections in the Red Hat Enterprise Linux 7 Linux Domain Identity,
Authentication, and Policy Guide.

2. On the server side, use the sec= option to enable the wanted security flavors. To enable all
security flavors as well as non-cryptographic mounts:

/export *(sec=sys:krb5:krb5i:krb5p)

Valid security flavors to use with the sec= option are:

sys: no cryptographic protection, the default

krb5: authentication only

krb5i : integrity protection

krb5p: privacy protection

3. On the client side, add sec= krb5 (or sec= krb5i , or sec= krb5p, depending on the setup)
to the mount options:

# mount -o sec=krb5 server:/export /mnt

73
St orage Administ rat ion G uide

For information on how to configure a NFS client, see the Setting up a Kerberos-aware NFS
Client section in the Red Hat Enterprise Linux 7 Linux Domain Identity, Authentication, and Policy
Guide.

Although Red Hat recommends using IdM, Active D irectory (AD ) Kerberos servers are also supported.
For details, see the following Red Hat Knowledgebase article: How to set up NFS using Kerberos
authentication on RHEL 7 using SSSD and Active D irectory.

For more information, see the exports(5) and nfs(5) manual pages, and Section 8.5, Common NFS
Mount Options .

For further information on the R P C SEC _G SS framework, including how g sspro xy and rpc. g ssd
inter-operate, see the GSSD flow description.

8 .8 .2 .1 . NFS Se curit y wit h NFSv4

NFSv4 includes ACL support based on the Microsoft Windows NT model, not the POSIX model,
because of the Microsoft Windows NT model's features and wide deployment.

Another important security feature of NFSv4 is the removal of the use of the MO UNT protocol for
mounting file systems. The MO UNT protocol presented a security risk because of the way the protocol
processed file handles.

8.8.3. File Permissions

Once the NFS file system is mounted read/write by a remote host, the only protection each shared file
has is its permissions. If two users that share the same user ID value mount the same NFS file system,
they can modify each others' files. Additionally, anyone logged in as root on the client system can
use the su - command to access any files with the NFS share.

By default, access control lists (ACLs) are supported by NFS under Red Hat Enterprise Linux. Red
Hat recommends that this feature is kept enabled.

By default, NFS uses root squashing when exporting a file system. This sets the user ID of anyone
accessing the NFS share as the root user on their local machine to no bo d y. Root squashing is
controlled by the default option ro o t_sq uash; for more information about this option, refer to
Section 8.7.1, The /etc/expo rts Configuration File . If possible, never disable root squashing.

When exporting an NFS share as read-only, consider using the al l _sq uash option. This option
makes every user accessing the exported file system take the user ID of the nfsno bo d y user.

8.9. NFS and rpcbi nd

Note

The following section only applies to NFSv3 implementations that require the rpcbi nd service
for backward compatibility.

The rpcbi nd [1] utility maps RPC services to the ports on which they listen. RPC processes notify
rpcbi nd when they start, registering the ports they are listening on and the RPC program numbers
they expect to serve. The client system then contacts rpcbi nd on the server with a particular RPC
program number. The rpcbi nd service redirects the client to the proper port number so it can
communicate with the requested service.

74
 Chapt er 8 . Net work File Syst em (NFS)

Because RPC-based services rely on rpcbi nd to make all connections with incoming client
requests, rpcbi nd must be available before any of these services start.

The rpcbi nd service uses TCP wrappers for access control, and access control rules for rpcbi nd
affect all RPC-based services. Alternatively, it is possible to specify access control rules for each of
the NFS RPC daemons. The man pages for rpc. mo untd and rpc. statd contain information
regarding the precise syntax for these rules.

8.9.1. T roubleshoot ing NFS and rpcbi nd

Because rpcbi nd [1] provides coordination between RPC services and the port numbers used to
communicate with them, it is useful to view the status of current RPC services using rpcbi nd when
troubleshooting. The rpci nfo command shows each RPC-based service with port numbers, an
RPC program number, a version number, and an IP protocol type (TCP or UD P).

To make sure the proper NFS RPC-based services are enabled for rpcbi nd , issue the following
command:

# rpci nfo -p

Examp le 8.7. rpci nfo -p co mman d o u t p u t

The following is sample output from this command:

program vers proto port service

100021 1 udp 32774 nlockmgr
100021 3 udp 32774 nlockmgr
100021 4 udp 32774 nlockmgr
100021 1 tcp 34437 nlockmgr
100021 3 tcp 34437 nlockmgr
100021 4 tcp 34437 nlockmgr
100011 1 udp 819 rquotad
100011 2 udp 819 rquotad
100011 1 tcp 822 rquotad
100011 2 tcp 822 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100005 1 udp 836 mountd
100005 1 tcp 839 mountd
100005 2 udp 836 mountd
100005 2 tcp 839 mountd
100005 3 udp 836 mountd
100005 3 tcp 839 mountd

If one of the NFS services does not start up correctly, rpcbi nd will be unable to map RPC requests
from clients for that service to the correct port. In many cases, if NFS is not present in rpci nfo
output, restarting NFS causes the service to correctly register with rpcbi nd and begin working.

For more information and a list of options on rpci nfo , refer to its man page.

8.10. References

75
St orage Administ rat ion G uide

Administering an NFS server can be a challenge. Many options, including quite a few not mentioned
in this chapter, are available for exporting or mounting NFS shares. Consult the following sources for
more information.

Inst alled Document at ion

man mo unt Contains a comprehensive look at mount options for both NFS server and client
configurations.

man fstab Gives details for the format of the /etc/fstab file used to mount file systems at
boot-time.

man nfs Provides details on NFS-specific file system export and mount options.

man expo rts Shows common options used in the /etc/expo rts file when exporting NFS file
systems.

Useful Websit es

http://linux-nfs.org The current site for developers where project status updates can be viewed.

http://nfs.sourceforge.net/ The old home for developers which still contains a lot of useful
information.

http://www.citi.umich.edu/projects/nfsv4/linux/ An NFSv4 for Linux 2.6 kernel resource.

http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.111.4086 An excellent whitepaper on

the features and enhancements of the NFS Version 4 protocol.

Relat ed Books

Managing NFS and NIS by Hal Stern, Mike Eisler, and Ricardo Labiaga; O'Reilly & Associates
Makes an excellent reference guide for the many different NFS export and mount options
available.

NFS Illustrated by Brent Callaghan; Addison-Wesley Publishing Company Provides

comparisons of NFS to other network file systems and shows, in detail, how NFS communication
occurs.

[1] The rpcbi nd s ervic e rep lac es po rtmap , whic h was us ed in p revio us vers io ns o f Red Hat
Enterp ris e Linux to map RPC p ro g ram numb ers to IP ad d res s p o rt numb er c o mb inatio ns . Fo r mo re
info rmatio n, refer to Sec tio n 8 .1.1, Req uired Servic es .

76
 Chapt er 9 . FS- Cache

Chapter 9. FS-Cache
FS-Cache is a persistent local cache that can be used by file systems to take data retrieved from over
the network and cache it on local disk. This helps minimize network traffic for users accessing data
from a file system mounted over the network (for example, NFS).

The following diagram is a high-level illustration of how FS-Cache works:

Fig u re 9 .1. FS- C ach e O verview

FS-Cache is designed to be as transparent as possible to the users and administrators of a system.

Unlike cachefs on Solaris, FS-Cache allows a file system on a server to interact directly with a
client's local cache without creating an overmounted file system. With NFS, a mount option instructs
the client to mount the NFS share with FS-cache enabled.

FS-Cache does not alter the basic operation of a file system that works over the network - it merely
provides that file system with a persistent place in which it can cache data. For instance, a client can
still mount an NFS share whether or not FS-Cache is enabled. In addition, cached NFS can handle
files that won't fit into the cache (whether individually or collectively) as files can be partially cached
and do not have to be read completely up front. FS-Cache also hides all I/O errors that occur in the
cache from the client file system driver.

77
St orage Administ rat ion G uide

To provide caching services, FS-Cache needs a cache back-end. A cache back-end is a storage
driver configured to provide caching services (i.e. cachefi l es). In this case, FS-Cache requires a
mounted block-based file system that supports bmap and extended attributes (e.g. ext3) as its cache
back-end.

FS-Cache cannot arbitrarily cache any file system, whether through the network or otherwise: the
shared file system's driver must be altered to allow interaction with FS-Cache, data storage/retrieval,
and metadata setup and validation. FS-Cache needs indexing keys and coherency data from the
cached file system to support persistence: indexing keys to match file system objects to cache
objects, and coherency data to determine whether the cache objects are still valid.

Note

In Red Hat Enterprise Linux 7, cachefi l esd is not installed by default and will need to be
installed manually.

9.1. Performance Guarant ee

FS-Cache does not guarantee increased performance, however it ensures consistent performance by
avoiding network congestion. Using a cache back-end incurs a performance penalty: for example,
cached NFS shares add disk accesses to cross-network lookups. While FS-Cache tries to be as
asynchronous as possible, there are synchronous paths (e.g. reads) where this isn't possible.

For example, using FS-Cache to cache an NFS share between two computers over an otherwise
unladen GigE network will not demonstrate any performance improvements on file access. Rather,
NFS requests would be satisfied faster from server memory rather than from local disk.

The use of FS-Cache, therefore, is a compromise between various factors. If FS-Cache is being used
to cache NFS traffic, for instance, it may slow the client down a little, but massively reduce the network
and server loading by satisfying read requests locally without consuming network bandwidth.

9.2. Set t ing Up a Cache

Currently, Red Hat Enterprise Linux 7 only provides the cachefi l es caching back-end. The
cachefi l esd daemon initiates and manages cachefi l es. The /etc/cachefi l esd . co nf file
controls how cachefi l es provides caching services. To configure a cache back-end of this type,
the cachefi l esd package must be installed.

The first setting to configure in a cache back-end is which directory to use as a cache. To configure
this, use the following parameter:

$ d i r /path/to/cache

Typically, the cache back-end directory is set in /etc/cachefi l esd . co nf as

/var/cache/fscache, as in:

$ dir /var/cache/fscache

FS-Cache will store the cache in the file system that hosts /path/to/cache. On a laptop, it is
advisable to use the root file system (/) as the host file system, but for a desktop machine it would be
more prudent to mount a disk partition specifically for the cache.

78
 Chapt er 9 . FS- Cache

File systems that support functionalities required by FS-Cache cache back-end include the Red Hat
Enterprise Linux 7 implementations of the following file systems:

ext3 (with extended attributes enabled)

ext4

BTRFS

XFS

The host file system must support user-defined extended attributes; FS-Cache uses these attributes to
store coherency maintenance information. To enable user-defined extended attributes for ext3 file
systems (i.e. device), use:

# tune2fs -o user_xattr /d ev/device

Alternatively, extended attributes for a file system can be enabled at mount time, as in:

# mo unt /d ev/device /path/to/cache -o user_xattr

The cache back-end works by maintaining a certain amount of free space on the partition hosting the
cache. It grows and shrinks the cache in response to other elements of the system using up free
space, making it safe to use on the root file system (for example, on a laptop). FS-Cache sets defaults
on this behavior, which can be configured via cache cull limits. For more information about
configuring cache cull limits, refer to Section 9.4, Setting Cache Cull Limits .

Once the configuration file is in place, start up the cachefi l esd daemon:

# servi ce cachefi l esd start

To configure cachefi l esd to start at boot time, execute the following command as root:

# chkco nfi g cachefi l esd o n

9.3. Using t he Cache Wit h NFS

NFS will not use the cache unless explicitly instructed. To configure an NFS mount to use FS-Cache,
include the -o fsc option to the mo unt command:

# mo unt nfs-share: / /mount/point -o fsc

All access to files under /mount/point will go through the cache, unless the file is opened for direct
I/O or writing (refer to Section 9.3.2, Cache Limitations With NFS for more information). NFS indexes
cache contents using NFS file handle, not the file name; this means that hard-linked files share the
cache correctly.

Caching is supported in version 2, 3, and 4 of NFS. However, each version uses different branches
for caching.

9.3.1. Cache Sharing

There are several potential issues to do with NFS cache sharing. Because the cache is persistent,
blocks of data in the cache are indexed on a sequence of four keys:

79
St orage Administ rat ion G uide

Level 1: Server details

Level 2: Some mount options; security type; FSID ; uniquifier

Level 3: File Handle

Level 4: Page number in file

To avoid coherency management problems between superblocks, all NFS superblocks that wish to
cache data have unique Level 2 keys. Normally, two NFS mounts with same source volume and
options will share a superblock, and thus share the caching, even if they mount different directories
within that volume.

Examp le 9 .1. C ach e sh arin g

Take the following two mo unt commands:

mo unt ho me0 : /d i sk0 /fred /ho me/fred -o fsc

mo unt ho me0 : /d i sk0 /ji m /ho me/ji m -o fsc

Here, /ho me/fred and /ho me/ji m will likely share the superblock as they have the same
options, especially if they come from the same volume/partition on the NFS server (ho me0 ). Now,
consider the next two subsequent mount commands:

mo unt ho me0 : /d i sk0 /fred /ho me/fred -o fsc,rsi ze= 230

mo unt ho me0 : /d i sk0 /ji m /ho me/ji m -o fsc,rsi ze= 231

In this case, /ho me/fred and /ho me/ji m will not share the superblock as they have different
network access parameters, which are part of the Level 2 key. The same goes for the following
mount sequence:

mo unt ho me0 : /d i sk0 /fred /ho me/fred 1 -o fsc,rsi ze= 230

mo unt ho me0 : /d i sk0 /fred /ho me/fred 2 -o fsc,rsi ze= 231

Here, the contents of the two subtrees (/ho me/fred 1 and /ho me/fred 2) will be cached twice.

Another way to avoid superblock sharing is to suppress it explicitly with the no sharecache
parameter. Using the same example:

mo unt ho me0 : /d i sk0 /fred /ho me/fred -o no sharecache,fsc

mo unt ho me0 : /d i sk0 /ji m /ho me/ji m -o no sharecache,fsc

However, in this case only one of the superblocks will be permitted to use cache since there is
nothing to distinguish the Level 2 keys of ho me0 : /d i sk0 /fred and ho me0 : /d i sk0 /ji m. To
address this, add a unique identifier on at least one of the mounts, i.e. fsc= unique-identifier.
For example:

mo unt ho me0 : /d i sk0 /fred /ho me/fred -o no sharecache,fsc

mo unt ho me0 : /d i sk0 /ji m /ho me/ji m -o no sharecache,fsc= ji m

Here, the unique identifier ji m will be added to the Level 2 key used in the cache for /ho me/ji m.

9.3.2. Cache Limit at ions Wit h NFS

80
 Chapt er 9 . FS- Cache

Opening a file from a shared file system for direct I/O will automatically bypass the cache. This is
because this type of access must be direct to the server.

Opening a file from a shared file system for writing will not work on NFS version 2 and 3. The
protocols of these versions do not provide sufficient coherency management information for the client
to detect a concurrent write to the same file from another client.

As such, opening a file from a shared file system for either direct I/O or writing will flush the cached
copy of the file. FS-Cache will not cache the file again until it is no longer opened for direct I/O or
writing.

Furthermore, this release of FS-Cache only caches regular NFS files. FS-Cache will not cache
directories, symlinks, device files, FIFOs and sockets.

9.4 . Set t ing Cache Cull Limit s

The cachefi l esd daemon works by caching remote data from shared file systems to free space on
the disk. This could potentially consume all available free space, which could be bad if the disk also
housed the root partition. To control this, cachefi l esd tries to maintain a certain amount of free
space by discarding old objects (i.e. accessed less recently) from the cache. This behavior is known
as cache culling.

Cache culling is done on the basis of the percentage of blocks and the percentage of files available
in the underlying file system. There are six limits controlled by settings in
/etc/cachefi l esd . co nf:

b ru n N% ( p ercen t ag e o f b lo cks) , f ru n N% ( p ercen t ag e o f f iles)

If the amount of free space and the number of available files in the cache rises above both
these limits, then culling is turned off.

b cu ll N% ( p ercen t ag e o f b lo cks) , f cu ll N% ( p ercen t ag e o f f iles)

If the amount of available space or the number of files in the cache falls below either of
these limits, then culling is started.

b st o p N% ( p ercen t ag e o f b lo cks) , f st o p N% ( p ercen t ag e o f f iles)

If the amount of available space or the number of available files in the cache falls below
either of these limits, then no further allocation of disk space or files is permitted until culling
has raised things above these limits again.

The default value of N for each setting is as follows:

brun/frun - 10%

bcul l /fcul l - 7%

bsto p/fsto p - 3%

When configuring these settings, the following must hold true:

0 <= bsto p < bcul l < brun < 100

0 <= fsto p < fcul l < frun < 100

These are the percentages of available space and available files and do not appear as 100 minus
the percentage displayed by the d f program.

81
St orage Administ rat ion G uide

Important

Culling depends on both bxxx and fxxx pairs simultaneously; they can not be treated
separately.

9.5. St at ist ical Informat ion

FS-Cache also keeps track of general statistical information. To view this information, use:

cat /pro c/fs/fscache/stats

FS-Cache statistics includes information on decision points and object counters. For more details on
the statistics provided by FS-Cache, refer to the following kernel document:

/usr/share/d o c/kernel -
d o c-version/D o cumentati o n/fi l esystems/cachi ng /fscache. txt

9.6. References

For more information on cachefi l esd and how to configure it, refer to man cachefi l esd and
man cachefi l esd . co nf. The following kernel documents also provide additional information:

/usr/share/d o c/cachefi l esd -version-number/R EAD ME

/usr/share/man/man5/cachefi l esd . co nf. 5. g z

/usr/share/man/man8/cachefi l esd . 8. g z

For general information about FS-Cache, including details on its design constraints, available
statistics, and capabilities, refer to the following kernel document:

/usr/share/d o c/kernel -
d o c-version/D o cumentati o n/fi l esystems/cachi ng /fscache. txt

82
 P art II. St orage Administ rat ion

Part II. Storage Administration

The Storage Administration section starts with storage considerations for Red Hat Enterprise Linux 7.
Instructions regarding partitions, logical volume management, and swap partitions follow this. D isk
Quotas, RAID systems are next, followed by the functions of mount command, volume_key, and acls.
SSD tuning, write barriers, I/O limits and diskless systems follow this. The large chapter of Online
Storage is next, and finally device mapper multipathing and virtual storage to finish.

Use the following Table of Contents to explore these Storage Administration tasks.

83
St orage Administ rat ion G uide

Chapter 10. Storage Considerations During Installation

Many storage device and file system settings can only be configured at install time. Other settings,
such as file system type, can only be modified up to a certain point without requiring a reformat. As
such, it is prudent that you plan your storage configuration accordingly before installing Red Hat
Enterprise Linux 7.

This chapter discusses several considerations when planning a storage configuration for your
system. For actual installation instructions (including storage configuration during installation), refer
to the Installation Guide provided by Red Hat.

For information on what Red Hat officially supports with regards to size and storage limits, refer to the
article http://www.redhat.com/resourcelibrary/articles/articles-red-hat-enterprise-linux-6-technology-
capabilities-and-limits.

10.1. Special Considerat ions

This section enumerates several issues and factors to consider for specific storage configurations.

Separat e Part it ions for /home, /opt , /usr/local

If it is likely that you will upgrade your system in the future, place /ho me, /o pt, and /usr/l o cal on
a separate device. This will allow you to reformat the devices/file systems containing the operating
system while preserving your user and application data.

DASD and z FCP Devices on IBM Syst em Z

On the IBM System Z platform, D ASD and zFCP devices are configured via the Channel Command
Word (CCW) mechanism. CCW paths must be explicitly added to the system and then brought online.
For D ASD devices, this is simply means listing the device numbers (or device number ranges) as the
D ASD = parameter at the boot command line or in a CMS configuration file.

For zFCP devices, you must list the device number, logical unit number (LUN), and world wide port name
(WWPN). Once the zFCP device is initialized, it is mapped to a CCW path. The FC P _x= lines on the
boot command line (or in a CMS configuration file) allow you to specify this information for the
installer.

Encrypt ing Block Devices Using LUKS

Formatting a block device for encryption using LUKS/d m-crypt will destroy any existing formatting
on that device. As such, you should decide which devices to encrypt (if any) before the new system's
storage configuration is activated as part of the installation process.

St ale BIOS RAID Met adat a

Moving a disk from a system configured for firmware RAID without removing the RAID metadata from
the disk can prevent An aco n d a from correctly detecting the disk.

84
 Chapt er 1 0 . St orage Considerat ions During Inst allat ion

Warning

Removing/deleting RAID metadata from disk could potentially destroy any stored data. Red
Hat recommends that you back up your data before proceeding.

To delete RAID metadata from the disk, use the following command:

d mrai d -r -E /device/

For more information about managing RAID devices, refer to man d mrai d and Chapter 17,
Redundant Array of Independent Disks (RAID).

iSCSI Det ect ion and Configurat ion

For plug and play detection of iSCSI drives, configure them in the firmware of an iBFT boot-capable
network interface card (NIC). CHAP authentication of iSCSI targets is supported during installation.
However, iSNS discovery is not supported during installation.

FCoE Det ect ion and Configurat ion

For plug and play detection of Fibre Channel over Ethernet (FCoE) drives, configure them in the
firmware of an ED D boot-capable NIC.

DASD

Direct-access storage devices (D ASD ) cannot be added/configured during installation. Such devices
are specified in the CMS configuration file.

Block Devices wit h DIF/DIX Enabled

D IF/D IX is a hardware checksum feature provided by certain SCSI host bus adapters and block
devices. When D IF/D IX is enabled, errors will occur if the block device is used as a general-purpose
block device. Buffered I/O or mmap(2)-based I/O will not work reliably, as there are no interlocks in
the buffered write path to prevent buffered data from being overwritten after the D IF/D IX checksum has
been calculated.

This will cause the I/O to later fail with a checksum error. This problem is common to all block device
(or file system-based) buffered I/O or mmap(2) I/O, so it is not possible to work around these errors
caused by overwrites.

As such, block devices with D IF/D IX enabled should only be used with applications that use
O _D IR EC T . Such applications should use the raw block device. Alternatively, it is also safe to use
the XFS file system on a D IF/D IX enabled block device, as long as only O _D IR EC T I/O is issued
through the file system. XFS is the only file system that does not fall back to buffered I/O when doing
certain allocation operations.

The responsibility for ensuring that the I/O data does not change after the D IF/D IX checksum has
been computed always lies with the application, so only applications designed for use with
O _D IR EC T I/O and D IF/D IX hardware should use D IF/D IX.

85
St orage Administ rat ion G uide

Chapter 11. File System Check

Filesystems may be checked for consistency, and optionally repaired, with filesystem-specific
userspace tools. These tools are often referred to as fsck tools, where fsck is a shortened version
of file system check.

Note

These filesystem checkers only guarantee metadata consistency across the filesystem; they
have no awareness of the actual data contained within the filesystem and are not data
recovery tools.

Filesystem inconsistencies can occur for various reasons, including but not limited to hardware
errors, storage administration errors, and software bugs.

Before modern metadata-journaling filesystems became common, a filesystem check was required
any time a system crashed or lost power. This was because a filesystem update could have been
interrupted, leading to an inconsistent state. As a result, a filesystem check is traditionally run on
each filesystem listed in /etc/fstab at boot-time. For journaling filesystems, this is usually a very
short operation, because the filesystem's metadata journaling ensures consistence even after a
crash.

However, there are times when a filesystem inconsistency or corruption may occur, even for
journaling filesystems. When this happens, the filesystem checker must be used to repair the
filesystem. The following will provide best practices and other useful information when performing
this procedure.

Important

Red Hat does not recommended this unless the machine does not boot, the file system is
extremely large, or the file system is on remote storage. It is possible to disable file system
check at boot by setting the sixth field in /etc/fstab to 0.

11.1. Best Pract ices for fsck

Generally, running the filesystem check and repair tool can be expected to automatically repair at
least some of the inconsistencies it finds. In some cases, severely damaged inodes or directories may
be discarded if they cannot be repaired. Significant changes to the filesystem may occur. To ensure
that unexpected or undesirable changes are not permanently made, perform the following
precautionary steps:

D ry ru n

Most filesystem checkers have a mode of operation which checks but does not repair the
filesystem. In this mode, the checker will print any errors that it finds and actions that it
would have taken, without actually modifying the filesystem.

86
 Chapt er 1 1 . File Syst em Check

Note

Later phases of consistency checking may print extra errors as it discovers

inconsistencies which would have been fixed in early phases if it were running in
repair mode.

O p erat e f irst o n a f ilesyst em imag e

Most filesystems support the creation of a metadata image, a sparse copy of the filesystem
which contains only metadata. Because filesystem checkers operate only on metadata,
such an image can be used to perform a dry run of an actual filesystem repair, to evaluate
what changes would actually be made. If the changes are acceptable, the repair can then
be performed on the filesystem itself.

Note

Severely damaged filesystems may cause problems with metadata image creation.

Save a f ilesyst em imag e f o r su p p o rt in vest ig at io n s

A pre-repair filesystem metadata image can often be useful for support investigations if
there is a possibility that the corruption was due to a software bug. Patterns of corruption
present in the pre-repair image may aid in root-cause analysis.

O p erat e o n ly o n u n mo u n t ed f ilesyst ems

A filesystem repair must be run only on unmounted filesystems. The tool must have sole
access to the filesystem or further damage may result. Most filesystem tools enforce this
requirement in repair mode, although some only support check-only mode on a mounted
filesystem. If check-only mode is run on a mounted filesystem, it may find spurious errors
that would not be found when run on an unmounted filesystem.

D isk erro rs

Filesystem check tools cannot repair hardware problems. A filesystem must be fully
readable and writable if repair is to operate successfully. If a filesystem was corrupted due
to a hardware error, the filesystem must first be moved to a good disk, for example with the
d d (8) utility.

11.2. Filesyst em-Specific Informat ion for fsck

11.2.1. ext 2, ext 3, and ext 4

All of these filesytems use the e2fsck binary to perform filesystem checks and repairs. The filenames
fsck. ext2, fsck. ext3, and fsck. ext4 are hardlinks to this same binary. These binaries are run
automatically at boot time and their behavior differs based on the filesystem being checked and the
state of the filesystem.

A full filesystem check and repair is invoked for ext2, which is not a metadata journaling filesystem,
and for ext4 filesystems without a journal.

For ext3 and ext4 filesystems with metadata journaling, the journal is replayed in userspace and the
binary exited. This is the default action as journal replay ensures a consistent filesystem after a

87
St orage Administ rat ion G uide

crash.

If these filesystems encounter metadata inconsistencies while mounted, they will record this fact in the
filesystem super block. If e2fsck finds that a filesystem is marked with such an error e2fsck will
perform a full check after replaying the journal (if present).

e2fsck may ask for user input during the run if the -p option is not specified. The -p option tells
e2fsck to automatically do all repairs that may be done safely. If user intervention is required,
e2fsck will indicate the unfixed problem in its output and reflect this status in the exit code.

Commonly used e2fsck run-time options include:

-n

No-modify mode. Check-only operation.

-b su p erb lo ck

Specify block number of an alternate suprerblock if the primary one is damaged.

-f

Force full check even if the superblock has no recorded errors.

-j jo u rn al- d ev

Specify the external journal device, if any.

-p

Automatically repair or " preen" the filesystem with no user input.

-y

Assume an answer of " yes" to all questions.

All options for e2fsck are specified in the e2fsck(8) manual page.

The following five basic phases are performed by e2fsck while running:

1. Inode, block, and size checks.

2. D irectory structure checks.

3. D irectory connectivity checks.

4. Reference count checks.

5. Group summary info checks.

The e2i mag e(8) utility can be used to create a metadata image prior to repair for diagnostic or
testing purposes. The -r option should be used for testing purposes in order to create a sparse file
of the same size as the filesystem itself. e2fsck can then operate directly on the resulting file. The -Q
option should be specified if the image is to be archived or provided for diagnostic. This creates a
more compact file format suitable for transfer.

11.2.2. XFS

No repair is performed automatically at boot time. To initiate a filesystem check or repair, the
xfs_repai r tool is used.

88
 Chapt er 1 1 . File Syst em Check

Note

Although an fsck. xfs binary is present in the xfsprogs package, this is present only to
satisfy initscripts that look for an fsck. filesystem binary at boot time. fsck. xfs
immediately exits with an exit code of 0.

Another thing to be aware of is that older xfsprogs packages contain an xfs_check tool. This
tool is very slow and does not scale well for large filesystems. As such, it has been depreciated
in favor of xfs_repai r -n.

A clean log on a filesystem is required for xfs_repai r to operate. If the filesystem was not cleanly
unmounted, it should be mounted and unmounted prior to using xfs_repai r. If the log is corrupt
and cannot be replayed, the -L option may be used to zero the log.

Important

The -L option must only be used if the log cannot be replayed. The option discards all
metadata updates in the log and will result in further inconsistencies.

It is possible to run xfs_repai r in a dry run, check-only mode by using the -n option. No changes
will be made to the filesystem when this option is specified.

xfs_repai r takes very few options. Commonly used options include:

-n

No modify mode. Check-only operation.

-L

Z ero metadata log. Use only if log cannot be replayed with mount.

-m maxmem

Limit memory used during run to maxmem MB. 0 can be specified to obtain a rough estimate
of the minimum memory required.

-l lo g d ev

Specify the external log device, if present.

All options for xfs_repai r are specified in the xfs_repai r(8) manual page.

The following eight basic phases are performed by xfs_repai r while running:

1. Inode and inode blockmap (addressing) checks.

2. Inode allocation map checks.

3. Inode size checks.

4. D irectory checks.

5. Pathname checks.

89
St orage Administ rat ion G uide

6. Link count checks.

7. Freemap checks.

8. Super block checks.

These phases, as well as messages printed during operation, are documented in depth in the
xfs_repai r(8) manual page.

xfs_repai r is not interactive. All operations are performed automatically with no input from the
user.

If it is desired to create a metadata image prior to repair for diagnostic or testing purposes, the
xfs_metad ump(8) and xfs_md resto re(8) utilities may be used.

11.2.3. Bt rfs

The btrfsck tool is used to check and repair btrfs filesystems. This tool is still in early development
and may not detect or repair all types of filesystem corruption.

By default, btrfsck does not make changes to the filesystem; that is, it runs check-only mode by
default. If repairs are desired the --repai r option must be specified.

The following three basic phases are performed by btrfsck while running:

1. Extent checks.

2. Filesystem root checks.

3. Root reference count checks.

The btrfs-i mag e(8) utility can be used to create a metadata image prior to repair for diagnostic or
testing purposes.

90
 Chapt er 1 2 . Part it ions

Chapter 12. Partitions

With the p art ed utility, you can:

View the existing partition table.

Change the size of existing partitions.

Add partitions from free space or additional hard drives.

The parted package is installed by default on Red Hat Enterprise Linux 7. To start p art ed , log in as
root and enter the parted /dev/sda command at a shell prompt, where /dev/sda is the device name
for the drive to configure.

Manipulat ing Part it ions on Devices in Use

For a device to not be in use, none of the partitions on the device can be mounted, and no swap
space on the device can be enabled.

If you want to remove or resize a partition, the device on which that partition resides must not be in
use.

Creating a new partition on a device that is in use is possible, but not recommended.

Modifying t he Part it ion T able

Modifying the partition table while another partition on the same disk is in use is generally not
recommended because the kernel is not be able to reread the partition table. As a consequence,
changes are not applied to a running system. In the described situation, reboot the system, or use the
following command to make the system register new or modified partitions:

# partx --upd ate --nr partition-number disk

The easiest way to modify disks that are currently in use is to boot the system in rescue mode if the
partitions on the disk are impossible to unmount, for example in the case of a system disk. When
prompted to mount the file system, select Ski p.

If the drive does not contain any partitions in use, that is there are no system processes that use or
lock the file system from being unmounted, you can unmount the partitions with the umo unt
command and turn off all the swap space on the hard drive with the swapo ff command.

Table 12.1, parted commands contains a list of commonly used p art ed commands. The following
sections explain some of these commands and arguments in more detail.

Important

D o not use the p art ed utility to create file systems. Use the mkf s tool instead.

T ab le 12.1. parted co mman d s

C o mman d D escrip t io n
check minor-num Perform a simple check of the file system

91
St orage Administ rat ion G uide

C o mman d D escrip t io n
cp from to Copy file system from one partition to another;
from and to are the minor numbers of the
partitions
hel p D isplay list of available commands
mkl abel label Create a disk label for the partition table
mkfs minor-num file-system-type Create a file system of type file-system-type
mkpart part-type [fs-type] start-mb Make a partition without creating a new file
end-mb system
mkpartfs part-type fs-type start-mb Make a partition and create the specified file
end-mb system
mo ve minor-num start-mb end-mb Move the partition
name minor-num name Name the partition for Mac and PC98 disklabels
only
pri nt D isplay the partition table
q ui t Quit p art ed
rescue start-mb end-mb Rescue a lost partition from start-mb to end-mb
resi ze minor-num start-mb end-mb Resize the partition from start-mb to end-mb
rm minor-num Remove the partition
sel ect device Select a different device to configure
set minor-num flag state Set the flag on a partition; state is either on or off
to g g l e [NUMBER [FLAG] Toggle the state of FLAG on partition NUMBER
uni t UNIT Set the default unit to UNIT

12.1. Viewing t he Part it ion T able

After starting p art ed , use the pri nt command to view the partition table. A table similar to the
following one appears:

Examp le 12.1. Part it io n t ab le

Model: ATA ST3160812AS (scsi)

Disk /dev/sda: 160GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags

1 32.3kB 107MB 107MB primary ext3 boot
2 107MB 105GB 105GB primary ext3
3 105GB 107GB 2147MB primary linux-swap
4 107GB 160GB 52.9GB extended root
5 107GB 133GB 26.2GB logical ext3
6 133GB 133GB 107MB logical ext3
7 133GB 160GB 26.6GB logical lvm

The first line contains the disk type, manufacturer, model number and interface, and the second
line displays the disk label type. The remaining output below the fourth line shows the partition
table.

92
 Chapt er 1 2 . Part it ions

In the partition table, the Minor number is the partition number. For example, the partition with minor
number 1 corresponds to /d ev/sd a1. The Start and End values are in megabytes. Valid T ype are
metadata, free, primary, extended, or logical. The Fi l esystem is the file system type, which can be
any of the following:

ext2

ext3

fat16

fat32

hfs

jfs

linux-swap

ntfs

reiserfs

hp-ufs

sun-ufs

xfs

If a Fi l esystem of a device shows no value, this means that its file system type is unknown.

The Fl ag s column lists the flags set for the partition. Available flags are boot, root, swap, hidden,
raid, lvm, or lba.

Note

To select a different device without having to restart parted , use the sel ect command
followed by the device name (for example, /d ev/sd a). D oing so allows you to view or
configure the partition table of a device.

12.2. Creat ing a Part it ion

Warning

D o not attempt to create a partition on a device that is in use.

Pro ced u re 12.1. C reat in g a p art it io n

1. Before creating a partition, boot into rescue mode (or unmount any partitions on the device
and turn off any swap space on the device).

2. Start parted , where /d ev/sda is the device on which to create the partition:

93
St orage Administ rat ion G uide

# parted /d ev/sda

3. View the current partition table to determine if there is enough free space:

# pri nt

If there is not enough free space, you can resize an existing partition. Refer to Section 12.5, Resizing
a Partition with fdisk for details.

12.2.1. Making t he Part it ion

From the partition table, determine the start and end points of the new partition and what partition
type it should be. You can only have four primary partitions (with no extended partition) on a device.
If you need more than four partitions, you can have three primary partitions, one extended partition,
and multiple logical partitions within the extended. For an overview of disk partitions, refer to the
appendix An Introduction to Disk Partitions in the Red Hat Enterprise Linux 7 Installation Guide.

For example, to create a primary partition with an ext3 file system from 1024 megabytes until 2048
megabytes on a hard drive type the following command:

# mkpart pri mary 10 24 20 4 8

Note

If you use the mkpartfs command instead, the file system is created after the partition is
created. However, parted does not support creating an ext3 file system. Thus, if you wish to
create an ext3 file system, use mkpart and create the file system with the mkfs command as
described later.

The changes start taking place as soon as you press Enter, so review the command before
executing to it.

After creating the partition, use the pri nt command to confirm that it is in the partition table with the
correct partition type, file system type, and size. Also remember the minor number of the new partition
so that you can label any file systems on it. You should also view the output of cat
/pro c/parti ti o ns after parted is closed to make sure the kernel recognizes the new partition.

The maximum number of partitions parted will create is 128. While the GUID Partition Table (GPT)
specification allows for more partitions by growing the area reserved for the partition table, common
practice used by parted is to limit it to enough area for 128 partitions.

12.2.2. Format t ing and Labeling t he Part it ion

To format and label the partition use the following procedure:

Pro ced u re 12.2. Fo rmat an d lab el t h e p art it io n

1. The partition does not have a file system. To create the ext4 file system, use:

# mkfs. ext4 /d ev/sda6

94
 Chapt er 1 2 . Part it ions

Warning

Formatting the partition permanently destroys any data that currently exists on the
partition.

2. Next, give the file system on the partition a label. For example, if the file system on the new
partition is /d ev/sd a6 and you want to label it /wo rk, use:

# e2l abel /dev/sda6 /work

By default, the installation program uses the mount point of the partition as the label to make sure the
label is unique. You can use any label you want.

Afterwards, create a mount point (e.g. /wo rk) as root.

12.2.3. Add t o /etc/fstab

As root, edit the /etc/fstab file to include the new partition using the partition's UUID . Use the
command bl ki d -o l i st for a complete list of the partition's UUID , or bl ki d d evi ce for
individual device details.

The first column should contain UUID = followed by the file system's UUID . The second column
should contain the mount point for the new partition, and the next column should be the file system
type (for example, ext3 or swap). If you need more information about the format, read the man page
with the command man fstab.

If the fourth column is the word d efaul ts, the partition is mounted at boot time. To mount the
partition without rebooting, as root, type the command:

mo unt /wo rk

12.3. Removing a Part it ion

Warning

D o not attempt to remove a partition on a device that is in use.

Pro ced u re 12.3. R emo ve a p art it io n

1. Before removing a partition, boot into rescue mode (or unmount any partitions on the device
and turn off any swap space on the device).

2. Start parted , where /d ev/sda is the device on which to remove the partition:

# parted /d ev/sda

3. View the current partition table to determine the minor number of the partition to remove:

# pri nt

95
St orage Administ rat ion G uide

4. Remove the partition with the command rm. For example, to remove the partition with minor
number 3:

# rm 3

The changes start taking place as soon as you press Enter, so review the command before
committing to it.

5. After removing the partition, use the pri nt command to confirm that it is removed from the
partition table. You should also view the output of /pro c/parti ti o ns to make sure the
kernel knows the partition is removed.

# cat /pro c/parti ti o ns

6. The last step is to remove it from the /etc/fstab file. Find the line that declares the removed
partition, and remove it from the file.

12.4 . Set t ing a Part it ion T ype

The partition type, not to be confused with the file system type, is used by a running system only
rarely. However, the partition type matters to on-the-fly generators, such as systemd -g pt-auto -
g enerato r, which use the partition type to, for example, automatically identify and mount devices.

You can start the fd i sk utility and use the t command to set the partition type. The following
example shows how to change the partition type of the first partition to 0x83, default on Linux:

# fd i sk /d ev/sd c
Command (m for help): t
Selected partition 1
Partition type (type L to list all types): 83
Changed type of partition 'Linux LVM' to 'Linux'.

The parted utility provides some control of partition types by trying to map the partition type to
'flags', which is not convenient for end users. The parted utility can handle only certain partition
types, for example LVM or RAID . To remove, for example, the lvm flag from the first partition with
parted , use:

# parted /d ev/sd c ' set 1 l vm o ff'

For a list of commonly used partition types and hexadecimal numbers used to represent them, see the
Partition Types table in the Partitions: Turning One D rive Into Many appendix of the Red Hat
Enterprise Linux 7 Installation Guide.

12.5. Resiz ing a Part it ion wit h fdisk

The fd i sk utility allows you to create and manipulate GPT, MBR, Sun, SGI, and BSD partition
tables. On disks with a GUID Partition Table (GPT), using the parted utility is recommended, as
fd i sk GPT support is in an experimental phase.

Before resizing a partition, back up the data stored on the file system and test the procedure, as the
only way to change a partition size using fd i sk is by deleting and recreating the partition.

96
 Chapt er 1 2 . Part it ions

Important

The partition you are resizing must be the last partition on a particular disk.

Red Hat only supports extending and resizing LVM partitions.

Pro ced u re 12.4 . R esiz in g a p art it io n

The following procedure is provided only for reference. To resize a partition using fd i sk:

1. Unmount the device:

~]# umount /dev/vda

2. Run fd i sk disk_name. For example:

~]# fdisk /dev/vda

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write

them. Be careful before using the write command.

Command (m for help):

3. Use the p option to determine the line number of the partition to be deleted.

Command (m for help): p

Disk /dev/vda: 16.1 GB, 16106127360 bytes, 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0006d09a

Device Boot Start End Blocks Id System

/dev/vda1 * 2048 1026047 512000 83 Linux
/dev/vda2 1026048 31457279 15215616 8e Linux LVM

4. Use the d option to delete a partition. If there is more than one partition available, fd i sk
prompts you to provide a number of the partition to delete:

Command (m for help): d

Partition number (1,2, default 2): 2
Partition 2 is deleted

5. Use the n option to create a partition and follow the prompts. Allow enough space for any
future resizing. The fd i sk default behavior (press Enter) is to use all space on the device.
You can specify the end of the partition by sectors, or specify a human-readable size by
using + <size><suffix>, for example +500M, or +10G.

97
St orage Administ rat ion G uide

Red Hat recommends using the human-readable size specification if you do not want to use
all free space, as fd i sk aligns the end of the partition with the physical sectors. If you
specify the size by providing an exact number (in sectors), fd i sk does not align the end of
the partition.

Command (m for help): n

Partition type:
p primary (1 primary, 0 extended, 3 free)
e extended
Select (default p): *Enter*
Using default response p
Partition number (2-4, default 2): *Enter*
First sector (1026048-31457279, default 1026048): *Enter*
Using default value 1026048
Last sector, +sectors or +size{K,M,G} (1026048-31457279, default
31457279): +500M
Partition 2 of type Linux and of size 500 MiB is set

6. Set the partition type to LVM:

Command (m for help): t

Partition number (1,2, default 2): *Enter*
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

7. Write the changes with the w option when you are sure the changes are correct, as errors can
cause instability with the selected partition.

8. Run e2fsck on the device to check for consistency:

~]# e2fsck /dev/vda

e2fsck 1.41.12 (17-May-2010)
Pass 1:Checking inodes, blocks, and sizes
Pass 2:Checking directory structure
Pass 3:Checking directory connectivity
Pass 4:Checking reference counts
Pass 5:Checking group summary information
ext4-1:11/131072 files (0.0% non-contiguous),27050/524128 blocks

9. Mount the device:

~]# mount /dev/vda

For more information, see the fdisk(8) manual page.

98
 Chapt er 1 3. Creat ing and Maint aining Snapshot s wit h Snapper

Chapter 13. Creating and Maintaining Snapshots with Snapper

A snapshot volume is a point in time copy of a target volume that provides a way to revert a file
system back to an earlier state. Snapper is a command-line tool to create and maintain snapshots for
Btrfs and thinly-provisioned LVM file systems.

13.1. Init ial Snapper Set up

Snapper requires discrete configuration files for each volume it operates on. You must set up the
configuration files manually. By default, only the root user is allowed to perform snapper commands.

Red Hat recommends using the ext4 file system with Snapper on Red Hat Enterprise Linux 7. Use the
XFS file system on lvm-thin volumes only if you are monitoring the amount of free space in the pool to
prevent out-of-space problems that can lead to a failure.

Note that the Btrfs tools and file system are provided as a Technology Preview, which make them
unsuitable for production systems.

Although it is possible to allow a user or group other than root to use certain Snapper commands,
Red Hat recommends that you do not add elevated permissions to otherwise unprivileged users or
groups. Such a configuration bypasses SELinux and could pose a security risk. Red Hat
recommends that you review these capabilities with your Security Team and consider using the sud o
infrastructure instead.

Pro ced u re 13.1. C reat in g a Sn ap p er C o n f ig u rat io n File

1. Create or choose either:

A thinly-provisioned logical volume with a Red Hat supported file system on top of it, or

A Btrfs subvolume.

2. Mount the file system.

3. Create the configuration file that defines this volume.

For LVM2:

# snapper -c config_name create-config -f "lvm(fs_type)" /mount-

point

For Btrfs:

# snapper -c config_name create-config -f btrfs /mount-point

The -c config_name option specifies the name of the configuration file.

The create-co nfi g tells snapper to create a configuration file.

The -f file_system tells snapper what file system to use; if this is omitted snapper will
attempt to detect the file system.

The /mount-point is where the subvolume or thinly-provisioned LVM2 file system is

mounted.

99
St orage Administ rat ion G uide

For example, to create a configuration file called l vm_co nfi g on an LVM2 subvolume with
an ext4 file system, mounted at /l vm_mo unt, use:

# snapper -c lvm_config create-config -f "lvm(ext4)" /lvm_mount

Alternatively, to create a configuration file called btrfs_co nfi g , on a Btrfs subvolume that
is mounted at /btrfs_mo unt, use:

# snapper -c btrfs_config create-config -f btrfs /btrfs_mount

The configuration files are stored in the /etc/snapper/co nfi g s/ directory.

13.2. Creat ing a Snapper Snapshot

Snapper can create the following kinds of snapshots:

Pre Sn ap sh o t

A pre snapshot serves as a point of origin for a post snapshot. The two are closely tied and
designed to track file system modification between the two points. The pre snapshot must be
created before the post snapshot.

Po st Sn ap sh o t

A post snapshot serves as the end point to the pre snapshot. The coupled pre and post
snapshots define a range for comparison. By default, every new snapper volume is
configured to create a background comparison after a related post snapshot is created
successfully.

Sin g le Sn ap sh o t

A single snapshot is a standalone snapshot created at a specific moment. These can be

used to track a timeline of modifications and have a general point to return to later.

13.2.1. Creat ing a Pre and Post Snapshot Pair

1 3.2 .1 .1 . Cre at ing a Pre Snapsho t wit h Snappe r

To create a pre snapshot, use:

# snapper -c config_name create -t pre

The -c config_name option creates a snapshot according to the specifications in the named
configuration file. If the configuration file does not yet exist, see Section 13.1, Initial Snapper Setup .

The create -t option specifies what type of snapshot to create. Accepted entries are pre, po st, or
si ng l e.

For example, to create a pre snapshot using the l vm_co nfi g configuration file, as created in
Section 13.1, Initial Snapper Setup , use:

# snapper -c SnapperExample create -t pre -p

1

The -p option prints the number of the created snapshot and is optional.

100
 Chapt er 1 3. Creat ing and Maint aining Snapshot s wit h Snapper

1 3.2 .1 .2 . Cre at ing a Po st Snapsho t wit h Snappe r

A post snapshot is the end point of the snapshot and should be created after the parent pre
snapshot by following the instructions in Section 13.2.1.1, Creating a Pre Snapshot with Snapper .

Pro ced u re 13.2. C reat in g a Po st Sn ap sh o t

1. D etermine the number of the pre snapshot:

# snapper -c config_name list

For example, to display the list of snapshots created using the configuration file
l vm_co nfi g , use the following:

# snapper -c lvm_config list

Type | # | Pre # | Date | User | Cleanup |
Description | Userdata
-------+---+-------+-------------------+------+----------+--------
-----+---------
single | 0 | | | root | | current
|
pre | 1 | | Mon 06<...> | root | |
|

The output above shows that the pre snapshot is number 1.

2. Create a post snapshot that is linked to a previously created pre snapshot:

# snapper -c config_file create -t post --pre-num

pre_snapshot_number

The -t po st option specifies the creation of the post snapshot type.

The --pre-num option specifies the corresponding pre snapshot.

For example, to create a post snapshot using the l vm_co nfi g configuration file and is
linked to pre snapshot number 1, use:

# snapper -c lvm_config create -t post --pre-num 1 -p

2

The -p option prints the number of the created snapshot and is optional.

3. The pre and post snapshots 1 and 2 are now created and paired. Verify this with the l i st
command:

# snapper -c lvm_config list

Type | # | Pre # | Date | User | Cleanup |
Description | Userdata
-------+---+-------+-------------------+------+----------+--------
-----+---------
single | 0 | | | root | | current
|

101
St orage Administ rat ion G uide

pre | 1 | | Mon 06<...> | root | |

|
post | 2 | 1 | Mon 06<...> | root | |
|

1 3.2 .1 .3. Wrapping a Co m m and in Pre and Po st Snapsho t s

You can also wrap a command within a pre and post snapshot, which can be useful when testing.
See Procedure 13.3, Wrapping a Command in Pre and Post Snapshots , which is a shortcut for the
following steps:

1. Running the snapper create pre snapsho t command.

2. Running a command or a list of commands to perform actions with a possible impact on the
file system content.

3. Running the snapper create po st snapsho t command.

Pro ced u re 13.3. Wrap p in g a C o mman d in Pre an d Po st Sn ap sh o t s

1. To wrap a command in pre and post snapshots:

# snapper -c lvm_config create --command "command_to_be_tracked"

For example, to track the creation of the /l vm_mo unt/hel l o _fi l e file:

# snapper -c lvm_config create --command "echo Hello >

/lvm_mount/hello_file"

2. To verify this, use the status command:

# snapper -c config_file status

first_snapshot_number..second_snapshot_number

For example, to track the changes made in the first step:

# snapper -c lvm_config status 3..4

+..... /lvm_mount/hello_file

Use the l i st command to verify the number of the snapshot if needed.

For more information on the status command, see Section 13.3, Tracking Changes
Between Snapper Snapshots .

Note that there is no guarantee that the command in the above example is the only thing the
snapshots capture. Snapper also records anything that is modified by the system, not just what a
user modifies.

13.2.2. Creat ing a Single Snapper Snapshot

Creating a single snapper snapshot is similar to creating a pre or post snapshot, only the create -t
option specifies single. The single snapshot is used to create a single snapshot in time without
having it relate to any others. However, if you are interested in a straightforward way to create
snapshots of LVM2 thin volumes without the need to automatically generate comparisons or list

102
 Chapt er 1 3. Creat ing and Maint aining Snapshot s wit h Snapper

additional information, Red Hat recommends using the System Storage Manager instead of Snapper
for this purpose, as described in Section 15.2.6, Snapshot .

To create a single snapshot, use:

# snapper -c config_name create -t single

For example, the following command creates a single snapshot using the l vm_co nfi g
configuration file.

# snapper -c lvm_config create -t single

Although single snapshots are not specifically designed to track changes, you can use the snapper
d i ff, xad i ff, and status commands to compare any two snapshots. For more information on
these commands, see Section 13.3, Tracking Changes Between Snapper Snapshots .

13.2.3. Configuring Snapper t o T ake Aut omat ed Snapshot s

Taking automated snapshots is one of key features of Snapper. By default, when you configure
Snapper for a volume, Snapper starts taking a snapshot of the volume every hour.

Under the default configuration, Snapper keeps:

10 hourly snapshots, and the final hourly snapshot is saved as a daily snapshot.

10 daily snapshots, and the final daily snapshot for a month is saved as a monthly snapshot.

10 monthly snapshots, and the final monthly snapshot is saved as a yearly snapshot.

10 yearly snapshots.

Note that Snapper keeps by default no more that 50 snapshots in total. However, Snapper keeps by
default all snapshots created less than 1,800 seconds ago.

The default configuration is specified in the /etc/snapper/co nfi g -templ ates/d efaul t file.
When you use the snapper create-co nfi g command to create a configuration, any unspecified
values are set based on the default configuration. You can edit the configuration for any defined
volume in the /etc/snapper/co nfi g s/config_name file.

13.3. T racking Changes Bet ween Snapper Snapshot s

Use the status, d i ff, and xad i ff commands to track the changes made to a subvolume between
snapshots:

st at u s

The status command shows a list of files and directories that have been created, modified,
or deleted between two snapshots, that is a comprehensive list of changes between two
snapshots. You can use this command to get an overview of the changes without excessive
details.

For more information, see Section 13.3.1, Comparing Changes with the status
Command .

d if f

103
St orage Administ rat ion G uide

The d i ff command shows a diff of modified files and directories between two snapshots
as received from the status command if there is at least one modification detected.

For more information, see Section 13.3.2, Comparing changes with the d i ff command .

xad if f

The xad i ff command compares how the extended attributes of a file or directory have
changed between two snapshots.

For more information, see Section 13.3.3, Comparing changes with the xad i ff
command .

13.3.1. Comparing Changes wit h t he status Command

The status command shows a list of files and directories that have been created, modified, or
deleted between two snapshots.

To display the status of files between two snapshots, use:

# snapper -c config_file status

first_snapshot_number..second_snapshot_number

Use the l i st command to determine snapshot numbers if needed.

For example, the following command displays the changes made between snapshot 1 and 2, using
the configuration file l vm_co nfi g .

snapper -c lvm_config status 1..2

tp.... /lvm_mount/dir1
-..... /lvm_mount/dir1/file_a
c.ug.. /lvm_mount/file2
+..... /lvm_mount/file3
....x. /lvm_mount/file4
cp..xa /lvm_mount/file5

Read letters and dots in the first part of the output as columns:

+..... /lvm_mount/file3
||||||
123456

Column 1 indicates any modification of the file (directory entry) type. Possible values are:

C o lu mn 1

Output Mean in g
. Nothing has changed.
+ File created.
- File deleted.
c Content changed.
t The type of directory entry has changed. For
example, a former symbolic link has changed to
a regular file with the same file name.

104
 Chapt er 1 3. Creat ing and Maint aining Snapshot s wit h Snapper

Column 2 indicates any changes in the file permissions. Possible values are:

C o lu mn 2

Output Mean in g
. No permissions changed.
p Permissions changed.

Column 3 indicates any changes in the user ownership. Possible values are:

C o lu mn 3

Output Mean in g
. No user ownership changed.
u User ownership has changed.

Column 4 indicates any changes in the group ownership. Possible values are:

C o lu mn 4

Output Mean in g
. No group ownership changed.
g Group ownership has changed.

Column 5 indicates any changes in the extended attributes. Possible values are:

C o lu mn 5

Output Mean in g
. No extended attributes changed.
x Extended attributes changed.

Column 6 indicates any changes in the access control lists (ACLs). Possible values are:

C o lu mn 6

Output Mean in g
. No ACLs changed.
a ACLs modified.

13.3.2. Comparing changes wit h t he d i ff command

The d i ff command shows the changes of modified files and directories between two snapshots.

# snapper -c config_name diff

first_snapshot_number..second_snapshot_number

Use the l i st command to determine the number of the snapshot if needed.

For example, to compare the changes made in files between snapshot 1 and snapshot 2 that were
made using the l vm_co nfi g configuration file, use:

105
St orage Administ rat ion G uide

# snapper -c lvm_config diff 1..2

--- /lvm_mount/.snapshots/13/snapshot/file4 19<...>
+++ /lvm_mount/.snapshots/14/snapshot/file4 20<...>
@ @ -0,0 +1 @ @
+words

The above output shows that fi l e4 had been modified to add " words" into the file.

13.3.3. Comparing changes wit h t he xad i ff command

The xad i ff command compares how the extended attributes of a file or directory have changed
between two snapshots:

# snapper -c config_name xadiff

first_snapshot_number..second_snapshot_number

Use the l i st command to determine the number of the snapshot if needed.

For example, to show the xadiff output between snapshot number 1 and snapshot number 2 that were
made using the l vm_co nfi g configuration file, use:

# snapper -c lvm_config xadiff 1..2

13.4 . Reversing Changes in Bet ween Snapshot s

To reverse changes made between two existing Snapper snapshots, use the und o chang e command
in the following format, where 1 is the first snapshot and 2 is the second snapshot:

snapper -c config_name und o chang e 1. . 2

Important

Using the und o chang e command does not revert the Snapper volume back to its original
state and does not provide data consistency. Any file modification that occurs outside of the
specified range, for example after snapshot 2, will remain unchanged after reverting back, for
example to the state of snapshot 1. For example, if und o chang e is run to undo the creation of
a user, any files owned by that user can still remain.

There is also no mechanism to ensure file consistency as a snapshot is made, so any

inconsistencies that already exist can be transferred back to the snapshot when the
und o chang e command is used.

D o not use the Snapper und o chang e command with the root file system, as doing so is likely
to lead to a failure.

The following diagram demonstrates how the und o chang e command works:

106
 Chapt er 1 3. Creat ing and Maint aining Snapshot s wit h Snapper

Fig u re 13.1. Sn ap p er St at u s O ver T ime

The diagram shows the point in time in which snapsho t_1 is created, fi l e_a is created, then
fi l e_b deleted. Snapsho t_2 is then created, after which fi l e_a is edited and fi l e_c is created.
This is now the current state of the system. The current system has an edited version of fi l e_a, no
fi l e_b, and a newly created fi l e_c.

When the und o chang e command is called, Snapper generates a list of modified files between the
first listed snapshot and the second. In the diagram, if you use the snapper -c SnapperExampl e
und o chang e 1. . 2 command, Snapper creates a list of modified files (that is, fi l e_a is created;
fi l e_b is deleted) and applies them to the current system. Therefore:

the current system will not have fi l e_a, as it has yet to be created when snapsho t_1 was
created.

fi l e_b will exist, copied from snapsho t_1 into the current system.

fi l e_c will exist, as its creation was outside the specified time.

Be aware that if fi l e_b and fi l e_c conflict, the system can become corrupted.

You can also use the snapper -c SnapperExampl e und o chang e 2. . 1 command. In this case,
the current system replaces the edited version of fi l e_a with one copied from snapsho t_1, which
undoes edits of that file made after snapsho t_2 was created.

Using t he mount and unmount Commands t o Reverse Changes

The und o chang e command is not always the best way to revert modifications. With the status and
d i ff command, you can make a qualified decision, and use the mo unt and unmo unt commands
instead of Snapper. The mo unt and unmo unt commands are only useful if you want to mount
snapshots and browse their content independently of Snapper workflow.

If needed, the mo unt command activates respective LVM Snapper snapshot before mounting. Use the
mo unt and unmo unt commands if you are, for example, interested in mounting snapshots and
extracting older version of several files manually. To revert files manually, copy them from a mounted

107
St orage Administ rat ion G uide

snapshot to the current file system. The current file system, snapshot 0, is the live file system created
in Procedure 13.1, Creating a Snapper Configuration File . Copy the files to the subtree of the
original /mount-point.

Use the mo unt and unmo unt commands for explicit client-side requests. The
/etc/snapper/co nfi g s/config_name file contains the ALLOW_USERS= and ALLOW_GROUPS=
variables where you can add users and groups. Then, snapperd allows you to perform mount
operations for the added users and groups.

13.5. Delet ing a Snapper Snapshot

To delete a snapshot:

# snapper -c config_name delete snapshot_number

You can use the l i st command to verify that the snapshot was successfully deleted.

108
 Chapt er 1 4 . Swap Space

Chapter 14. Swap Space

Swap space in Linux is used when the amount of physical memory (RAM) is full. If the system needs
more memory resources and the RAM is full, inactive pages in memory are moved to the swap space.
While swap space can help machines with a small amount of RAM, it should not be considered a
replacement for more RAM. Swap space is located on hard drives, which have a slower access time
than physical memory. Swap space can be a dedicated swap partition (recommended), a swap file,
or a combination of swap partitions and swap files. Note that Btrfs does not support swap space.

In years past, the recommended amount of swap space increased linearly with the amount of RAM in
the system. However, modern systems often include hundreds of gigabytes of RAM. As a
consequence, recommended swap space is considered a function of system memory workload, not
system memory.

Table 14.1, Recommended System Swap Space provides the recommended size of a swap partition
depending on the amount of RAM in your system and whether you want sufficient memory for your
system to hibernate. The recommended swap partition size is established automatically during
installation. To allow for hibernation, however, you need to edit the swap space in the custom
partitioning stage.

Recommendations in Table 14.1, Recommended System Swap Space are especially important on
systems with low memory (1 GB and less). Failure to allocate sufficient swap space on these systems
can cause issues such as instability or even render the installed system unbootable.

T ab le 14 .1. R eco mmen d ed Syst em Swap Sp ace

Amo u n t o f R AM in t h e R eco mmen d ed swap sp ace R eco mmen d ed swap sp ace

syst em if allo win g f o r h ib ern at io n
2 GB 2 times the amount of RAM 3 times the amount of RAM
> 2 GB 8 GB Equal to the amount of RAM 2 times the amount of RAM
> 8 GB 64 GB At least 4 GB 1.5 times the amount of RAM
> 64 GB At least 4 GB Hibernation not recommended

At the border between each range listed in Table 14.1, Recommended System Swap Space , for
example a system with 2 GB, 8 GB, or 64 GB of system RAM, discretion can be exercised with regard
to chosen swap space and hibernation support. If your system resources allow for it, increasing the
swap space may lead to better performance. A swap space of at least 100 GB is recommended for
systems with over 140 logical processors or over 3 TB of RAM.

Note that distributing swap space over multiple storage devices also improves swap space
performance, particularly on systems with fast drives, controllers, and interfaces.

Important

File systems and LVM2 volumes assigned as swap space should not be in use when being
modified. Any attempts to modify swap fail if a system process or the kernel is using swap
space. Use the free and cat /pro c/swaps commands to verify how much and where swap
is in use.

You should modify swap space while the system is booted in rescue mode, see Booting Your
Computer in Rescue Mode in the Red Hat Enterprise Linux 7 Installation Guide. When prompted to
mount the file system, select Ski p.

109
St orage Administ rat ion G uide

14 .1. Adding Swap Space

Sometimes it is necessary to add more swap space after installation. For example, you may upgrade
the amount of RAM in your system from 1 GB to 2 GB, but there is only 2 GB of swap space. It might
be advantageous to increase the amount of swap space to 4 GB if you perform memory-intense
operations or run applications that require a large amount of memory.

You have three options: create a new swap partition, create a new swap file, or extend swap on an
existing LVM2 logical volume. It is recommended that you extend an existing logical volume.

14 .1.1. Ext ending Swap on an LVM2 Logical Volume

By default, Red Hat Enterprise Linux 7 uses all available space during installation. If this is the case
with your system, then you must first add a new physical volume to the volume group used by the
swap space.

After adding additional storage to the swap space's volume group, it is now possible to extend it. To
do so, perform the following procedure (assuming /d ev/Vo l G ro up0 0 /Lo g Vo l 0 1 is the volume
you want to extend by 2 GB):

Pro ced u re 14 .1. Ext en d in g Swap o n an LVM2 Lo g ical Vo lu me

1. D isable swapping for the associated logical volume:

# swapoff -v /dev/VolGroup00/LogVol01

2. Resize the LVM2 logical volume by 2 GB:

# lvresize /dev/VolGroup00/LogVol01 -L +2G

3. Format the new swap space:

# mkswap /dev/VolGroup00/LogVol01

4. Enable the extended logical volume:

# swapon -v /dev/VolGroup00/LogVol01

To test if the logical volume was successfully extended, use cat /pro c/swaps or free to inspect
the swap space.

14 .1.2. Creat ing an LVM2 Logical Volume for Swap

To add a swap volume group (assuming /d ev/Vo l G ro up0 0 /Lo g Vo l 0 2 is the swap volume you
want to add):

1. Create the LVM2 logical volume of size 2 GB:

# l vcreate Vo l G ro up0 0 -n Lo g Vo l 0 2 -L 2G

2. Format the new swap space:

# mkswap /d ev/Vo l G ro up0 0 /Lo g Vo l 0 2

110
 Chapt er 1 4 . Swap Space

3. Add the following entry to the /etc/fstab file:

# /d ev/Vo l G ro up0 0 /Lo g Vo l 0 2 swap swap d efaul ts 0 0

4. Enable the extended logical volume:

# swapo n -v /d ev/Vo l G ro up0 0 /Lo g Vo l 0 2

To test if the logical volume was successfully created, use cat /pro c/swaps or free to inspect the
swap space.

14 .1.3. Creat ing a Swap File

To add a swap file:

Pro ced u re 14 .2. Ad d a swap f ile

1. D etermine the size of the new swap file in megabytes and multiply by 1024 to determine the
number of blocks. For example, the block size of a 64 MB swap file is 65536.

2. At a shell, type the following command with co unt being equal to the desired block size:

# d d i f= /d ev/zero o f= /swapfi l e bs= 10 24 co unt= 6 5536

3. Setup the swap file with the command:

# mkswap /swapfi l e

4. Change the security of the swapfile so it is not world readable.

# chmo d 0 6 0 0 /swapfi l e

5. To enable the swap file immediately but not automatically at boot time:

# swapo n /swapfi l e

6. To enable it at boot time, edit /etc/fstab to include the following entry:

/swapfi l e swap swap d efaul ts 0 0

The next time the system boots, it enables the new swap file.

To test if the new swap file was successfully created, use cat /pro c/swaps or free to inspect the
swap space.

14 .2. Removing Swap Space

Sometimes it can be prudent to reduce swap space after installation. For example, say you
downgraded the amount of RAM in your system from 1 GB to 512 MB, but there is 2 GB of swap
space still assigned. It might be advantageous to reduce the amount of swap space to 1 GB, since
the larger 2 GB could be wasting disk space.

111
St orage Administ rat ion G uide

You have three options: remove an entire LVM2 logical volume used for swap, remove a swap file, or
reduce swap space on an existing LVM2 logical volume.

14 .2.1. Reducing Swap on an LVM2 Logical Volume

To reduce an LVM2 swap logical volume (assuming /d ev/Vo l G ro up0 0 /Lo g Vo l 0 1 is the volume
you want to reduce):

Pro ced u re 14 .3. R ed u cin g an LVM2 swap lo g ical vo lu me

1. D isable swapping for the associated logical volume:

# swapo ff -v /d ev/Vo l G ro up0 0 /Lo g Vo l 0 1

2. Reduce the LVM2 logical volume by 512 MB:

# l vred uce /d ev/Vo l G ro up0 0 /Lo g Vo l 0 1 -L -512M

3. Format the new swap space:

# mkswap /d ev/Vo l G ro up0 0 /Lo g Vo l 0 1

4. Enable the extended logical volume:

# swapo n -v /d ev/Vo l G ro up0 0 /Lo g Vo l 0 1

To test if the swap's logical volume size was successfully reduced, use cat /pro c/swaps or free
to inspect the swap space.

14 .2.2. Removing an LVM2 Logical Volume for Swap

To remove a swap volume group (assuming /d ev/Vo l G ro up0 0 /Lo g Vo l 0 2 is the swap volume
you want to remove):

Pro ced u re 14 .4 . R emo ve a swap vo lu me g ro u p

1. D isable swapping for the associated logical volume:

# swapo ff -v /d ev/Vo l G ro up0 0 /Lo g Vo l 0 2

2. Remove the LVM2 logical volume of size 512 MB:

# l vremo ve /d ev/Vo l G ro up0 0 /Lo g Vo l 0 2

3. Remove the following entry from the /etc/fstab file:

/d ev/Vo l G ro up0 0 /Lo g Vo l 0 2 swap swap d efaul ts 0 0

To test if the logical volume size was successfully removed, use cat /pro c/swaps or free to
inspect the swap space.

14 .2.3. Removing a Swap File

112
 Chapt er 1 4 . Swap Space

To remove a swap file:

Pro ced u re 14 .5. R emo ve a swap f ile

1. At a shell prompt, execute the following command to disable the swap file (where /swapfi l e
is the swap file):

# swapo ff -v /swapfi l e

2. Remove its entry from the /etc/fstab file.

3. Remove the actual file:

# rm /swapfi l e

14 .3. Moving Swap Space

To move swap space from one location to another, follow the steps for removing swap space, and
then follow the steps for adding swap space.

113
St orage Administ rat ion G uide

Chapter 15. System Storage Manager (SSM)

System Storage Manager (SSM) provides a command line interface to manage storage in various
technologies. Storage systems are becoming increasingly complicated through the use of D evice
Mappers (D M), Logical Volume Managers (LVM), and Multiple D evices (MD ). This creates a system
that is not user friendly and makes it easier for errors and problems to arise. SSM alleviates this by
creating a unified user interface. This interface allows users to run complicated systems in a simple
manner. For example, to create and mount a new file system without SSM, there are five commands
that must be used. With SSM only one is needed.

This chapter will explain how SSM interacts with various back ends, then detail some common use
cases.

15.1. SSM Backends

SSM uses a core abstraction layer in ssml i b/mai n. py which complies with the device, pool, and
volume abstraction, ignoring the specifics of the underlying technology. Backends can be registered
in ssml i b/mai n. py to handle specific storage technology methods, such as create, snapsho t,
or to remo ve volumes and pools.

There are already several backends registered in SSM. The following sections will give basic
information on them as well as definitions on how they handle pools, volumes, snapshots, and
devices.

15.1.1. BT RFS Backend

BTRFS, a file system with many advanced features, is used as a volume management backend in
SSM. Pools, volumes, and snapshots can be created with the BTRFS backend.

1 5 .1 .1 .1 . BT RFS Po o l

The BTRFS file system itself is the pool. It can be extended by adding more devices or shrunk by
removing devices. SSM creates a BTRFS file system when a BTRFS pool is created. This means that
every new BTRFS pool has one volume of the same name as the pool which cannot be removed
without removing the entire pool. The default BTRFS pool name is btrfs_po o l .

The name of the pool is used as the file system label. If there is already an existing BTRFS file system
in the system without a label, the BTRFS pool will generate a name for internal use in the format of
btrfs_device_base_name.

1 5 .1 .1 .2 . BT RFS Vo lum e

Volumes created after the first volume in a pool are the same as sub-volumes. SSM will temporarily
mount the BTRFS file system if it is unmounted in order to create a sub-volume.

The name of a volume is used as the subvolume path in the BTRFS file system. For example, a
subvolume displays as /d ev/l vm_po o l /l vo l 0 0 1. Every object in this path must exist in order for
the volume to be created. Volumes can also be referenced with its mount point.

1 5 .1 .1 .3. BT RFS Snapsho t

Snapshots can be taken of any BTRFS volume in the system with SSM. Be aware that BTRFS does
not distinguish between subvolumes and snapshots. While this means that SSM cannot recognize
the BTRFS snapshot destination, it will try to recognize special name formats. If the name specified

114
 Chapt er 1 5. Syst em St orage Manager (SSM)

when creating the snapshot does the specific pattern, the snapshot will not be recognized and
instead be listed as a regular BTRFS volume.

1 5 .1 .1 .4 . BT RFS De vice

BTRFS does not require any special device to be created on.

15.1.2. LVM Backend

Pools, volumes, and snapshots can be created with LVM. The following definitions are from an LVM
point of view.

1 5 .1 .2 .1 . LVM Po o l

LVM pool is the same as an LVM volume group. This means that grouping devices and new logical
volumes can be created out of the LVM pool. The default LVM pool name is l vm_po o l .

1 5 .1 .2 .2 . LVM Vo lum e

An LVM volume is the same as an ordinary logical volume.

1 5 .1 .2 .3. LVM Snapsho t

When a snapshot is created from the LVM volume a new snapsho t volume is created which can then
be handled just like any other LVM volume. Unlike BTRFS, LVM is able to distinguish snapshots from
regular volumes so there is no need for a snapshot name to match a particular pattern.

1 5 .1 .2 .4 . LVM De vice

SSM makes the need for an LVM backend to be created on a physical device transparent for the user.

15.1.3. Crypt

The crypt backend in SSM uses cryptsetup and d m-crypt targ et to manage encrypted volumes.
Crypt backends can be used as a regular backend for creating encrypted volumes on top of regular
block devices (or on other volumes such as LVM or MD volumes), or to create encrypted LVM
volumes in a single steps.

Only volumes can be created with a crypt backend; pooling is not supported and it does not require
special devices.

The following sections define volumes and snapshots from the crypt point of view.

1 5 .1 .3.1 . Crypt Vo lum e

Crypt volumes are created by d m-crypt and represent the data on the original encrypted device in
an unencrypted form. It does not support RAID or any device concatenation.

Two modes, or extensions, are supported: luks and plain. Luks is used by default. For more
information on the extensions, refer to man cryptsetup.

1 5 .1 .3.2 . Crypt Snapsho t

While the crypt backend does not support snapshotting, if the encrypted volume is created on top of
an LVM volume, the volume itself can be snapshotted. The snapshot can then be opened by using

115
St orage Administ rat ion G uide

cryptsetup.

15.1.4 . Mult iple Devices (MD)

MD backend is currently limited to only gathering the information about MD volumes in the system.

15.2. Common SSM T asks

The following sections will go through basic use cases covering how to install SSM then display
information about all detected devices, pools, and volumes. Next, a pool will be created with two
volumes and an XFS file system. The file system will then be checked for consistency, then a volume
will be increased in size. Then a snapshot will be created. Finally one of the volumes will be removed.

15.2.1. SSM Inst allat ion

To install SSM use the following command:

# yum install system-storage-manager

There are several backends that are enabled only if the supporting packages are installed:

The LVM backend requires the l vm2 package.

The BTRFS backend requires the btrfs-pro g s package.

The Crypt backend requires the d evi ce-mapper and cryptsetup packages.

15.2.2. Show informat ion about all det ect ed devices

D isplaying information about all detected devices, pools, volumes, and snapshots is done with the
l i st command. Running the ssm l i st with no options will display the following output:

~]# ssm list

----------------------------------------------------------
Device Free Used Total Pool Mount point
----------------------------------------------------------
/dev/sda 2.00 GB PARTITIONED
/dev/sda1 47.83 MB /test
/dev/vda 15.00 GB PARTITIONED
/dev/vda1 500.00 MB /boot
/dev/vda2 0.00 KB 14.51 GB 14.51 GB rhel
----------------------------------------------------------
------------------------------------------------
Pool Type Devices Free Used Total
------------------------------------------------
rhel lvm 1 0.00 KB 14.51 GB 14.51 GB
------------------------------------------------
------------------------------------------------------------------------
---------
Volume Pool Volume size FS FS size Free Type
Mount point
------------------------------------------------------------------------
---------
/dev/rhel/root rhel 13.53 GB xfs 13.52 GB 9.64 GB linear /

116
 Chapt er 1 5. Syst em St orage Manager (SSM)

/dev/rhel/swap rhel 1000.00 MB linear

/dev/sda1 47.83 MB xfs 44.50 MB 44.41 MB part
/test
/dev/vda1 500.00 MB xfs 496.67 MB 403.56 MB part
/boot
------------------------------------------------------------------------
---------

This display can be further narrowed down by using arguments to specify what should be displayed.
The list of available options can be found with the ssm l i st --hel p command.

Note

D epending on the argument given, SSM may not display everything.

Running the d evi ces or d ev argument will omit some devices. CD Roms and D M/MD
devices, for example, are intentionally hidden as they are listed as volumes.
Some backends do not support snapshots and cannot distinguish between a snapshot
and a regular volume. Running the snapsho t argument on one of these backends will
cause SSM to attempt to recognize the volume name in order to identify a snapshot. If the
SSM regular expression does not match the snapshot pattern then the snapshot will not be
recognized.
With the exception of the main BTRFS volume (the file system itself), any unmounted
BTRFS volumes will not be shown.

15.2.3. Creat e a new pool, logical volume, and file syst em

In this section, a new pool will be created with a default name. It will have the devices /d ev/vd b and
/d ev/vd c, a logical volume of 1G, and an XFS file system.

The command to create this scenario is ssm create --fs xfs -s 1G /d ev/vd b /d ev/vd c.
The following options are used:

The --fs option specifies the required file system type. Current supported file system types are:

ext3

ext4

xfs

btrfs

The -s specifies the size of the logical volume. The following suffixes are supported to define
units:

K or k for kilobytes

M or m for megabytes

G or g for gigabytes

T or t for terabytes

P or p for petabytes

117
St orage Administ rat ion G uide

E or e for exabytes

The two listed devices, /d ev/vd b and /d ev/vd c, are the two devices I wish to create.

~]# ssm create --fs xfs -s 1G /dev/vdb /dev/vdc

Physical volume "/dev/vdb" successfully created
Physical volume "/dev/vdc" successfully created
Volume group "lvm_pool" successfully created
Logical volume "lvol001" created

There are two other options for the ssm co mmand that may be useful. The first is the -p pool
command. This specifies the pool the volume is to be created on. If it does not yet exist, then SSM will
create it. This was omitted in the above example which caused SSM to use the default name
l vm_po o l . However, to use a specific name to fit in with any existing naming conventions, the -p
option should be used.

The second useful option is the -n name command. This names the newly created logical volume.
As with the -p, this is needed in order to use a specific name to fit in with any existing naming
conventions.

An example of these two options being used follows:

~]# ssm create --fs xfs -p new_pool -n XFS_Volume /dev/vdd

Volume group "new_pool" successfully created
Logical volume "XFS_Volume" created

SSM has now created two physical volumes, a pool, and a logical volume with the ease of only one
command.

15.2.4 . Check a file syst em's consist ency

The ssm check command checks the file system consistency on the volume. It is possible to specify
multiple volumes to check. If there is no file system on the volume, then the volume will be skipped.

To check all devices in the volume l vo l 0 0 1, run the command ssm check
/d ev/l vm_po o l /l vo l 0 0 1.

~]# ssm check /dev/lvm_pool/lvol001

Checking xfs file system on '/dev/mapper/lvm_pool-lvol001'.
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...

118
 Chapt er 1 5. Syst em St orage Manager (SSM)

- check for inodes claiming duplicate blocks...

- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

15.2.5. Increase a volume's siz e

The ssm resi ze command changes the size of the specified volume and file system. If there is no
file system then only the volume itself will be resized.

For this example, we currently have one logical volume on /d ev/vd b that is 900MB called
l vo l 0 0 1.

~]# ssm list

-----------------------------------------------------------------
Device Free Used Total Pool Mount point
-----------------------------------------------------------------
/dev/vda 15.00 GB PARTITIONED
/dev/vda1 500.00 MB /boot
/dev/vda2 0.00 KB 14.51 GB 14.51 GB rhel
/dev/vdb 120.00 MB 900.00 MB 1.00 GB lvm_pool
/dev/vdc 1.00 GB
-----------------------------------------------------------------
---------------------------------------------------------
Pool Type Devices Free Used Total
---------------------------------------------------------
lvm_pool lvm 1 120.00 MB 900.00 MB 1020.00 MB
rhel lvm 1 0.00 KB 14.51 GB 14.51 GB
---------------------------------------------------------
------------------------------------------------------------------------
--------------------
Volume Pool Volume size FS FS size Free
Type Mount point
------------------------------------------------------------------------
--------------------
/dev/rhel/root rhel 13.53 GB xfs 13.52 GB 9.64 GB
linear /
/dev/rhel/swap rhel 1000.00 MB
linear
/dev/lvm_pool/lvol001 lvm_pool 900.00 MB xfs 896.67 MB 896.54 MB
linear

119
St orage Administ rat ion G uide

/dev/vda1 500.00 MB xfs 496.67 MB 403.56 MB

part /boot
------------------------------------------------------------------------
--------------------

The logical volume needs to be increased by another 500MB. To do so we will need to add an extra
device to the pool:

~]# ssm resize -s +500M /dev/lvm_pool/lvol001 /dev/vdc

Physical volume "/dev/vdc" successfully created
Volume group "lvm_pool" successfully extended
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
Extending logical volume lvol001 to 1.37 GiB
Logical volume lvol001 successfully resized
meta-data=/dev/mapper/lvm_pool-lvol001 isize=256 agcount=4,
agsize=57600 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0
data = bsize=4096 blocks=230400, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=853, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 230400 to 358400

SSM runs a check on the device and then extends the volume by the specified amount. This can be
verified with the ssm l i st command.

120
 Chapt er 1 5. Syst em St orage Manager (SSM)

~]# ssm list

------------------------------------------------------------------
Device Free Used Total Pool Mount point
------------------------------------------------------------------
/dev/vda 15.00 GB PARTITIONED
/dev/vda1 500.00 MB /boot
/dev/vda2 0.00 KB 14.51 GB 14.51 GB rhel
/dev/vdb 0.00 KB 1020.00 MB 1.00 GB lvm_pool
/dev/vdc 640.00 MB 380.00 MB 1.00 GB lvm_pool
------------------------------------------------------------------
------------------------------------------------------
Pool Type Devices Free Used Total
------------------------------------------------------
lvm_pool lvm 2 640.00 MB 1.37 GB 1.99 GB
rhel lvm 1 0.00 KB 14.51 GB 14.51 GB
------------------------------------------------------
------------------------------------------------------------------------
----------------------
Volume Pool Volume size FS FS size
Free Type Mount point
------------------------------------------------------------------------
----------------------
/dev/rhel/root rhel 13.53 GB xfs 13.52 GB 9.64 GB
linear /
/dev/rhel/swap rhel 1000.00 MB
linear
/dev/lvm_pool/lvol001 lvm_pool 1.37 GB xfs 1.36 GB 1.36 GB
linear
/dev/vda1 500.00 MB xfs 496.67 MB 403.56 MB
part /boot
------------------------------------------------------------------------
----------------------

Note

It is only possible to decrease an LVM volume's size; it is not supported with other volume
types. This is done by using a - instead of a + . For example, to decrease the size of an LVM
volume by 50M the command would be:

~]# ssm resize -s-50M /dev/lvm_pool/lvol002

Rounding size to boundary between physical extents: 972.00 MiB
WARNING: Reducing active logical volume to 972.00 MiB
THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce lvol002? [y/n]: y
Reducing logical volume lvol002 to 972.00 MiB
Logical volume lvol002 successfully resized

Without either the + or -, the value is taken as absolute.

15.2.6. Snapshot

121
St orage Administ rat ion G uide

To take a snapshot of an existing volume, use the ssm snapsho t command.

Note

This operation will fail if the back end the volume belongs to does not support snapshotting.

To create a snapshot of the l vo l 0 0 1, use the following command:

~]# ssm snapshot /dev/lvm_pool/lvol001

Logical volume "snap20150519T130900" created

To verify this, use the ssm l i st, and note the extra snapshot section.

~]# ssm list

----------------------------------------------------------------
Device Free Used Total Pool Mount point
----------------------------------------------------------------
/dev/vda 15.00 GB PARTITIONED
/dev/vda1 500.00 MB /boot
/dev/vda2 0.00 KB 14.51 GB 14.51 GB rhel
/dev/vdb 0.00 KB 1020.00 MB 1.00 GB lvm_pool
/dev/vdc 1.00 GB
----------------------------------------------------------------
--------------------------------------------------------
Pool Type Devices Free Used Total
--------------------------------------------------------
lvm_pool lvm 1 0.00 KB 1020.00 MB 1020.00 MB
rhel lvm 1 0.00 KB 14.51 GB 14.51 GB
--------------------------------------------------------
------------------------------------------------------------------------
----------------------
Volume Pool Volume size FS FS size
Free Type Mount point
------------------------------------------------------------------------
----------------------
/dev/rhel/root rhel 13.53 GB xfs 13.52 GB 9.64 GB
linear /
/dev/rhel/swap rhel 1000.00 MB
linear
/dev/lvm_pool/lvol001 lvm_pool 900.00 MB xfs 896.67 MB 896.54 MB
linear
/dev/vda1 500.00 MB xfs 496.67 MB 403.56 MB
part /boot
------------------------------------------------------------------------
----------------------
------------------------------------------------------------------------
----------
Snapshot Origin Pool Volume size
Size Type
------------------------------------------------------------------------
----------

122
 Chapt er 1 5. Syst em St orage Manager (SSM)

/dev/lvm_pool/snap20150519T130900 lvol001 lvm_pool 120.00 MB 0.00

KB linear
------------------------------------------------------------------------
----------

15.2.7. Remove an it em

The ssm remo ve is used to remove an item, either a device, pool, or volume.

Note

If a device is being used by a pool when removed, it will fail. This can be forced using the -f
argument.

If the volume is mounted when removed, it will fail. Unlike the device, it cannot be forced with
the -f argument.

To remove the l vm_po o l and everything within it use the following command:

~]# ssm remove lvm_pool

Do you really want to remove volume group "lvm_pool" containing 2
logical volumes? [y/n]: y
Do you really want to remove active logical volume snap20150519T130900?
[y/n]: y
Logical volume "snap20150519T130900" successfully removed
Do you really want to remove active logical volume lvol001? [y/n]: y
Logical volume "lvol001" successfully removed
Volume group "lvm_pool" successfully removed

15.3. SSM Resources

Further reading on SSM can be found with the following resources:

The man ssm page provides good descriptions and examples, as well as details on all of the
commands and options too specific to be documented here.

Local documentation for SSM is stored in the d o c/ directory.

The SSM wiki can be accessed at http://storagemanager.sourceforge.net/index.html.

The mailing list can be subscribed to from

https://lists.sourceforge.net/lists/listinfo/storagemanager-devel and mailing list archives from
http://sourceforge.net/mailarchive/forum.php?forum_name=storagemanager-devel. The mailing list
is where developers communicate. There is currently no user mailing list so feel free to post
questions there as well.

123
St orage Administ rat ion G uide

Chapter 16. Disk Quotas

D isk space can be restricted by implementing disk quotas which alert a system administrator before a
user consumes too much disk space or a partition becomes full.

D isk quotas can be configured for individual users as well as user groups. This makes it possible to
manage the space allocated for user-specific files (such as email) separately from the space
allocated to the projects a user works on (assuming the projects are given their own groups).

In addition, quotas can be set not just to control the number of disk blocks consumed but to control
the number of inodes (data structures that contain information about files in UNIX file systems).
Because inodes are used to contain file-related information, this allows control over the number of
files that can be created.

The q uo ta RPM must be installed to implement disk quotas.

Note

This chapter is for all file systems, however some file systems have their own quota
management tools. Refer to the corresponding description for the applicable file systems.

For XFS file systems, refer to Section 3.3, XFS Quota Management .

Btrfs does not have disk quotas so is not covered.

16.1. Configuring Disk Quot as

To implement disk quotas, use the following steps:

1. Enable quotas per file system by modifying the /etc/fstab file.

2. Remount the file system(s).

3. Create the quota database files and generate the disk usage table.

4. Assign quota policies.

Each of these steps is discussed in detail in the following sections.

16.1.1. Enabling Quot as

As root, using a text editor, edit the /etc/fstab file.

Examp le 16 .1. Ed it /etc/fstab

For example, to use the text editor vi m type the following:

# vim /etc/fstab

Add the usrq uo ta and/or g rpq uo ta options to the file systems that require quotas:

124
 Chapt er 1 6 . Disk Q uot as

Examp le 16 .2. Ad d q u o t as

/dev/VolGroup00/LogVol00 / ext3 defaults 1 1

LABEL=/boot /boot ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/VolGroup00/LogVol02 /home ext3 defaults,usrquota,grpquota
1 2
/dev/VolGroup00/LogVol01 swap swap defaults 0 0 . . .

In this example, the /ho me file system has both user and group quotas enabled.

Note

The following examples assume that a separate /ho me partition was created during the
installation of Red Hat Enterprise Linux. The root (/) partition can be used for setting quota
policies in the /etc/fstab file.

16.1.2. Remount ing t he File Syst ems

After adding the usrq uo ta and/or g rpq uo ta options, remount each file system whose fstab entry
has been modified. If the file system is not in use by any process, use one of the following methods:

Issue the umo unt command followed by the mo unt command to remount the file system. Refer to
the man page for both umo unt and mo unt for the specific syntax for mounting and unmounting
various file system types.

Issue the mo unt -o remo unt file-system command (where file-system is the name of the
file system) to remount the file system. For example, to remount the /ho me file system, the
command to issue is mo unt -o remo unt /ho me.

If the file system is currently in use, the easiest method for remounting the file system is to reboot the
system.

16.1.3. Creat ing t he Quot a Dat abase Files

After each quota-enabled file system is remounted run the q uo tacheck command.

The q uo tacheck command examines quota-enabled file systems and builds a table of the current
disk usage per file system. The table is then used to update the operating system's copy of disk
usage. In addition, the file system's disk quota files are updated.

Note

The q uo tacheck command has no effect on XFS as the table of disk usage is completed
automatically at mount time. Refer to the man page xfs_q uo ta(8) for more information.

125
St orage Administ rat ion G uide

To create the quota files (aq uo ta. user and aq uo ta. g ro up) on the file system, use the -c option
of the q uo tacheck command.

Examp le 16 .3. C reat e q u o t a f iles

For example, if user and group quotas are enabled for the /ho me file system, create the files in the
/ho me directory:

# q uo tacheck -cug /ho me

The -c option specifies that the quota files should be created for each file system with quotas
enabled, the -u option specifies to check for user quotas, and the -g option specifies to check for
group quotas.

If neither the -u or -g options are specified, only the user quota file is created. If only -g is specified,
only the group quota file is created.

After the files are created, run the following command to generate the table of current disk usage per
file system with quotas enabled:

# q uo tacheck -avug

The options used are as follows:

Check all quota-enabled, locally-mounted file systems

D isplay verbose status information as the quota check proceeds

Check user disk quota information

Check group disk quota information

After q uo tacheck has finished running, the quota files corresponding to the enabled quotas (user
and/or group) are populated with data for each quota-enabled locally-mounted file system such as
/ho me.

16.1.4 . Assigning Quot as per User

The last step is assigning the disk quotas with the ed q uo ta command.

To configure the quota for a user, as root in a shell prompt, execute the command:

# ed q uo ta username

Perform this step for each user who needs a quota. For example, if a quota is enabled in
/etc/fstab for the /ho me partition (/d ev/Vo l G ro up0 0 /Lo g Vo l 0 2 in the example below) and
the command ed q uo ta testuser is executed, the following is shown in the editor configured as the
default for the system:

126
 Chapt er 1 6 . Disk Q uot as

Disk quotas for user testuser (uid 501):

Filesystem blocks soft hard inodes soft
hard
/dev/VolGroup00/LogVol02 440436 0 0 37418 0
0

Note

The text editor defined by the ED IT O R environment variable is used by ed q uo ta. To change
the editor, set the ED IT O R environment variable in your ~ /. bash_pro fi l e file to the full
path of the editor of your choice.

The first column is the name of the file system that has a quota enabled for it. The second column
shows how many blocks the user is currently using. The next two columns are used to set soft and
hard block limits for the user on the file system. The i no d es column shows how many inodes the
user is currently using. The last two columns are used to set the soft and hard inode limits for the
user on the file system.

The hard block limit is the absolute maximum amount of disk space that a user or group can use.
Once this limit is reached, no further disk space can be used.

The soft block limit defines the maximum amount of disk space that can be used. However, unlike the
hard limit, the soft limit can be exceeded for a certain amount of time. That time is known as the grace
period. The grace period can be expressed in seconds, minutes, hours, days, weeks, or months.

If any of the values are set to 0, that limit is not set. In the text editor, change the desired limits.

Examp le 16 .4 . C h an g e d esired limit s

For example:

Disk quotas for user testuser (uid 501):

Filesystem blocks soft hard inodes soft
hard
/dev/VolGroup00/LogVol02 440436 500000 550000 37418 0
0

To verify that the quota for the user has been set, use the command:

# quota username
Disk quotas for user username (uid 501):
Filesystem blocks quota limit grace files quota limit
grace
/dev/sdb 1000* 1000 1000 0 0 0

16.1.5. Assigning Quot as per Group

Quotas can also be assigned on a per-group basis. For example, to set a group quota for the d evel
group (the group must exist prior to setting the group quota), use the command:

127
St orage Administ rat ion G uide

# ed q uo ta -g d evel

This command displays the existing quota for the group in the text editor:

Disk quotas for group devel (gid 505):

Filesystem blocks soft hard inodes soft
hard
/dev/VolGroup00/LogVol02 440400 0 0 37418 0
0

Modify the limits, then save the file.

To verify that the group quota has been set, use the command:

# q uo ta -g d evel

16.1.6. Set t ing t he Grace Period for Soft Limit s

If a given quota has soft limits, you can edit the grace period (i.e. the amount of time a soft limit can
be exceeded) with the following command:

# ed q uo ta -t

This command works on quotas for inodes or blocks, for either users or groups.

Important

While other ed q uo ta commands operate on quotas for a particular user or group, the -t
option operates on every file system with quotas enabled.

16.2. Managing Disk Quot as

If quotas are implemented, they need some maintenance mostly in the form of watching to see if the
quotas are exceeded and making sure the quotas are accurate.

Of course, if users repeatedly exceed their quotas or consistently reach their soft limits, a system
administrator has a few choices to make depending on what type of users they are and how much
disk space impacts their work. The administrator can either help the user determine how to use less
disk space or increase the user's disk quota.

16.2.1. Enabling and Disabling

It is possible to disable quotas without setting them to 0. To turn all user and group quotas off, use
the following command:

# q uo tao ff -vaug

If neither the -u or -g options are specified, only the user quotas are disabled. If only -g is specified,
only group quotas are disabled. The -v switch causes verbose status information to display as the
command executes.

128
 Chapt er 1 6 . Disk Q uot as

To enable quotas again, use the q uo tao n command with the same options.

For example, to enable user and group quotas for all file systems, use the following command:

# q uo tao n -vaug

To enable quotas for a specific file system, such as /ho me, use the following command:

# q uo tao n -vug /ho me

If neither the -u or -g options are specified, only the user quotas are enabled. If only -g is specified,
only group quotas are enabled.

Note

The q uo tao n command is not always needed for XFS because it is performed automatically
at mount time. Refer to the man page q uo tao n(8) for more information.

16.2.2. Report ing on Disk Quot as

Creating a disk usage report entails running the repq uo ta utility.

Examp le 16 .5. O u t p u t o f repq uo ta co mman d

For example, the command repq uo ta /ho me produces this output:

*** Report for user quotas on device /dev/mapper/VolGroup00-LogVol02

Block grace time: 7days; Inode grace time: 7days
Block limits File limits
User used soft hard grace used soft hard grace
----------------------------------------------------------------------
root -- 36 0 0 4 0 0
kristin -- 540 0 0 125 0 0
testuser -- 440400 500000 550000 37418 0 0

To view the disk usage report for all (option -a) quota-enabled file systems, use the command:

# repq uo ta -a

While the report is easy to read, a few points should be explained. The -- displayed after each user
is a quick way to determine whether the block or inode limits have been exceeded. If either soft limit is
exceeded, a + appears in place of the corresponding -; the first - represents the block limit, and the
second represents the inode limit.

The g race columns are normally blank. If a soft limit has been exceeded, the column contains a time
specification equal to the amount of time remaining on the grace period. If the grace period has
expired, no ne appears in its place.

16.2.3. Keeping Quot as Accurat e

129
St orage Administ rat ion G uide

When a file system fails to unmount cleanly (due to a system crash, for example), it is necessary to
run q uo tacheck. However, q uo tacheck can be run on a regular basis, even if the system has not
crashed. Safe methods for periodically running q uo tacheck include:

En su rin g q u o t ach eck ru n s o n n ext reb o o t

Note

This method works best for (busy) multiuser systems which are periodically rebooted.

As root, place a shell script into the /etc/cro n. d ai l y/ or /etc/cro n. weekl y/

directoryor schedule one using the cro ntab -e commandthat contains the to uch
/fo rceq uo tacheck command. This creates an empty fo rceq uo tacheck file in the root
directory, which the system init script looks for at boot time. If it is found, the init script runs
q uo tacheck. Afterward, the init script removes the /fo rceq uo tacheck file; thus,
scheduling this file to be created periodically with cro n ensures that q uo tacheck is run
during the next reboot.

For more information about cro n, refer to man cro n.

R u n n in g q u o t ach eck in sin g le u ser mo d e

An alternative way to safely run q uo tacheck is to boot the system into single-user mode to
prevent the possibility of data corruption in quota files and run the following commands:

# q uo tao ff -vug /file_system

# q uo tacheck -vug /file_system

# q uo tao n -vug /file_system

R u n n in g q u o t ach eck o n a ru n n in g syst em

If necessary, it is possible to run q uo tacheck on a machine during a time when no users

are logged in, and thus have no open files on the file system being checked. Run the
command q uo tacheck -vug file_system; this command will fail if q uo tacheck
cannot remount the given file_system as read-only. Note that, following the check, the file
system will be remounted read-write.

Warning

Running q uo tacheck on a live file system mounted read-write is not recommended

due to the possibility of quota file corruption.

Refer to man cro n for more information about configuring cro n.

16.3. Disk Quot a References

For more information on disk quotas, refer to the man pages of the following commands:

q uo tacheck

130
 Chapt er 1 6 . Disk Q uot as

ed q uo ta

repq uo ta

q uo ta

q uo tao n

q uo tao ff

131
St orage Administ rat ion G uide

Chapter 17. Redundant Array of Independent Disks (RAID)

The basic idea behind RAID is to combine multiple small, inexpensive disk drives into an array to
accomplish performance or redundancy goals not attainable with one large and expensive drive.
This array of drives appears to the computer as a single logical storage unit or drive.

RAID allows information to be spread across several disks. RAID uses techniques such as disk
striping (RAID Level 0), disk mirroring (RAID Level 1), and disk striping with parity (RAID Level 5) to
achieve redundancy, lower latency, increased bandwidth, and maximized ability to recover from hard
disk crashes.

RAID distributes data across each drive in the array by breaking it down into consistently-sized
chunks (commonly 256K or 512k, although other values are acceptable). Each chunk is then written
to a hard drive in the RAID array according to the RAID level employed. When the data is read, the
process is reversed, giving the illusion that the multiple drives in the array are actually one large
drive.

System Administrators and others who manage large amounts of data would benefit from using RAID
technology. Primary reasons to deploy RAID include:

Enhances speed

Increases storage capacity using a single virtual disk

Minimizes data loss from disk failure

17.1. RAID T ypes

There are three possible RAID approaches: Firmware RAID , Hardware RAID and Software RAID .

Firmware RAID

Firmware RAID (also known as ATARAID ) is a type of software RAID where the RAID sets can be
configured using a firmware-based menu. The firmware used by this type of RAID also hooks into the
BIOS, allowing you to boot from its RAID sets. D ifferent vendors use different on-disk metadata
formats to mark the RAID set members. The Intel Matrix RAID is a good example of a firmware RAID
system.

Hardware RAID

The hardware-based array manages the RAID subsystem independently from the host. It presents a
single disk per RAID array to the host.

A Hardware RAID device may be internal or external to the system, with internal devices commonly
consisting of a specialized controller card that handles the RAID tasks transparently to the operating
system and with external devices commonly connecting to the system via SCSI, Fibre Channel, iSCSI,
InfiniBand, or other high speed network interconnect and presenting logical volumes to the system.

RAID controller cards function like a SCSI controller to the operating system, and handle all the
actual drive communications. The user plugs the drives into the RAID controller (just like a normal
SCSI controller) and then adds them to the RAID controllers configuration. The operating system will
not be able to tell the difference.

Soft ware RAID

132
 Chapt er 1 7 . Redundant Array of Independent Disks (RAID)

Software RAID implements the various RAID levels in the kernel disk (block device) code. It offers the
cheapest possible solution, as expensive disk controller cards or hot-swap chassis [2] are not
required. Software RAID also works with cheaper ID E disks as well as SCSI disks. With today's faster
CPUs, Software RAID also generally outperforms Hardware RAID .

The Linux kernel contains a multi-disk (MD ) driver that allows the RAID solution to be completely
hardware independent. The performance of a software-based array depends on the server CPU
performance and load.

Here are some of the key features of the Linux software RAID stack:

Multi-threaded design

Portability of arrays between Linux machines without reconstruction

Backgrounded array reconstruction using idle system resources

Hot-swappable drive support

Automatic CPU detection to take advantage of certain CPU features such as streaming SIMD
support

Automatic correction of bad sectors on disks in an array

Regular consistency checks of RAID data to ensure the health of the array

Proactive monitoring of arrays with email alerts sent to a designated email address on important
events

Write-intent bitmaps which drastically increase the speed of resync events by allowing the kernel
to know precisely which portions of a disk need to be resynced instead of having to resync the
entire array

Resync checkpointing so that if you reboot your computer during a resync, at startup the resync
will pick up where it left off and not start all over again

The ability to change parameters of the array after installation. For example, you can grow a 4-
disk RAID 5 array to a 5-disk RAID 5 array when you have a new disk to add. This grow operation
is done live and does not require you to reinstall on the new array.

17.2. RAID Levels and Linear Support

RAID supports various configurations, including levels 0, 1, 4, 5, 6, 10, and linear. These RAID types
are defined as follows:

Level 0

RAID level 0, often called " striping," is a performance-oriented striped data mapping
technique. This means the data being written to the array is broken down into strips and
written across the member disks of the array, allowing high I/O performance at low inherent
cost but provides no redundancy.

Many RAID level 0 implementations will only stripe the data across the member devices up
to the size of the smallest device in the array. This means that if you have multiple devices
with slightly different sizes, each device will get treated as though it is the same size as the
smallest drive. Therefore, the common storage capacity of a level 0 array is equal to the

133
St orage Administ rat ion G uide

capacity of the smallest member disk in a Hardware RAID or the capacity of smallest
member partition in a Software RAID multiplied by the number of disks or partitions in the
array.

Level 1

RAID level 1, or " mirroring," has been used longer than any other form of RAID . Level 1
provides redundancy by writing identical data to each member disk of the array, leaving a
" mirrored" copy on each disk. Mirroring remains popular due to its simplicity and high level
of data availability. Level 1 operates with two or more disks, and provides very good data
reliability and improves performance for read-intensive applications but at a relatively high
cost. [3]

The storage capacity of the level 1 array is equal to the capacity of the smallest mirrored
hard disk in a Hardware RAID or the smallest mirrored partition in a Software RAID . Level 1
redundancy is the highest possible among all RAID types, with the array being able to
operate with only a single disk present.

Level 4

Level 4 uses parity [4] concentrated on a single disk drive to protect data. Because the
dedicated parity disk represents an inherent bottleneck on all write transactions to the RAID
array, level 4 is seldom used without accompanying technologies such as write-back
caching, or in specific circumstances where the system administrator is intentionally
designing the software RAID device with this bottleneck in mind (such as an array that will
have little to no write transactions once the array is populated with data). RAID level 4 is so
rarely used that it is not available as an option in Anaconda. However, it could be created
manually by the user if truly needed.

The storage capacity of Hardware RAID level 4 is equal to the capacity of the smallest
member partition multiplied by the number of partitions minus one. Performance of a RAID
level 4 array will always be asymmetrical, meaning reads will outperform writes. This is
because writes consume extra CPU and main memory bandwidth when generating parity,
and then also consume extra bus bandwidth when writing the actual data to disks because
you are writing not only the data, but also the parity. Reads need only read the data and
not the parity unless the array is in a degraded state. As a result, reads generate less traffic
to the drives and across the busses of the computer for the same amount of data transfer
under normal operating conditions.

Level 5

This is the most common type of RAID . By distributing parity across all of an array's
member disk drives, RAID level 5 eliminates the write bottleneck inherent in level 4. The only
performance bottleneck is the parity calculation process itself. With modern CPUs and
Software RAID , that is usually not a bottleneck at all since modern CPUs can generate
parity very fast. However, if you have a sufficiently large number of member devices in a
software RAID 5 array such that the combined aggregate data transfer speed across all
devices is high enough, then this bottleneck can start to come into play.

As with level 4, level 5 has asymmetrical performance, with reads substantially

outperforming writes. The storage capacity of RAID level 5 is calculated the same way as
with level 4.

Level 6

This is a common level of RAID when data redundancy and preservation, and not
performance, are the paramount concerns, but where the space inefficiency of level 1 is not
acceptable. Level 6 uses a complex parity scheme to be able to recover from the loss of any

134
 Chapt er 1 7 . Redundant Array of Independent Disks (RAID)

two drives in the array. This complex parity scheme creates a significantly higher CPU
burden on software RAID devices and also imposes an increased burden during write
transactions. As such, level 6 is considerably more asymmetrical in performance than levels
4 and 5.

The total capacity of a RAID level 6 array is calculated similarly to RAID level 5 and 4,
except that you must subtract 2 devices (instead of 1) from the device count for the extra
parity storage space.

Level 10

This RAID level attempts to combine the performance advantages of level 0 with the
redundancy of level 1. It also helps to alleviate some of the space wasted in level 1 arrays
with more than 2 devices. With level 10, it is possible to create a 3-drive array configured to
store only 2 copies of each piece of data, which then allows the overall array size to be 1.5
times the size of the smallest devices instead of only equal to the smallest device (like it
would be with a 3-device, level 1 array).

The number of options available when creating level 10 arrays (as well as the complexity of
selecting the right options for a specific use case) make it impractical to create during
installation. It is possible to create one manually using the command line md ad m tool. For
details on the options and their respective performance trade-offs, refer to man md .

Lin ear R AID

Linear RAID is a simple grouping of drives to create a larger virtual drive. In linear RAID , the
chunks are allocated sequentially from one member drive, going to the next drive only when
the first is completely filled. This grouping provides no performance benefit, as it is unlikely
that any I/O operations will be split between member drives. Linear RAID also offers no
redundancy and, in fact, decreases reliability if any one member drive fails, the entire
array cannot be used. The capacity is the total of all member disks.

17.3. Linux RAID Subsyst ems

RAID in Linux is composed of the following subsystems:

Linux Hardware RAID cont roller drivers

Hardware RAID controllers have no specific RAID subsystem in Linux. Because they use special
RAID chipsets, hardware RAID controllers come with their own drivers; these drivers allow the system
to detect the RAID sets as regular disks.

mdraid

The md rai d subsystem was designed as a software RAID solution for Linux; it is also the preferred
solution for software RAID under Linux. This subsystem uses its own metadata format, generally
referred to as native md rai d metadata.

md rai d also supports other metadata formats, known as external metadata. Red Hat Enterprise
Linux 7 uses md rai d with external metadata to access ISW / IMSM (Intel firmware RAID ) sets.
md rai d sets are configured and controlled through the md ad m utility.

dmraid

Device-mapper RAID or d mrai d refers to device-mapper kernel code that offers the mechanism to
piece disks together into a RAID set. This same kernel code does not provide any RAID configuration

135
St orage Administ rat ion G uide

mechanism.

d mrai d is configured entirely in user-space, making it easy to support various on-disk metadata
formats. As such, d mrai d is used on a wide variety of firmware RAID implementations. d mrai d also
supports Intel firmware RAID , although Red Hat Enterprise Linux 7 uses md rai d to access Intel
firmware RAID sets.

17.4 . RAID Support in t he Inst aller

The An aco n d a installer will automatically detect any hardware and firmware RAID sets on a system,
making them available for installation. An aco n d a also supports software RAID using md rai d , and
can recognize existing md rai d sets.

An aco n d a provides utilities for creating RAID sets during installation; however, these utilities only
allow partitions (as opposed to entire disks) to be members of new sets. To use an entire disk for a
set, simply create a partition on it spanning the entire disk, and use that partition as the RAID set
member.

When the root file system uses a RAID set, An aco n d a will add special kernel command-line options
to the bootloader configuration telling the i ni trd which RAID set(s) to activate before searching for
the root file system.

For instructions on configuring RAID during installation, refer to the Red Hat Enterprise Linux 7
Installation Guide.

17.5. Convert ing Root Disk t o RAID1 aft er Inst allat ion
If you need to convert a non-raided root disk to a RAID 1 mirror after installing Red Hat
Enterprise Linux 7, see the instructions in the following Red Hat Knowledgebase article: How do I
convert my root disk to RAID 1 after installation of Red Hat Enterprise Linux 7?

On the PowerPC (PPC) architecture, take the following additional steps:

1. Copy the contents of the PowerPC Reference Platform (PReP) boot partition from /d ev/sd a1
to /d ev/sd b1:

# dd if=/dev/sda1 of=/dev/sdb1

2. Update the Prep and boot flag on the first partition on both disks:

$ parted /dev/sda set 1 prep on

$ parted /dev/sda set 1 boot on

$ parted /dev/sdb set 1 prep on

$ parted /dev/sdb set 1 boot on

Note that running the g rub2-i nstal l /d ev/sd a command does not work on a PowerPC machine
and returns an error, but the system boots as expected.

17.6. Configuring RAID Set s

Most RAID sets are configured during creation, typically through the firmware menu or from the
installer. In some cases, you may need to create or modify RAID sets after installing the system,
preferably without having to reboot the machine and enter the firmware menu to do so.

136
 Chapt er 1 7 . Redundant Array of Independent Disks (RAID)

Some hardware RAID controllers allow you to configure RAID sets on-the-fly or even define
completely new sets after adding extra disks. This requires the use of driver-specific utilities, as there
is no standard API for this. Refer to your hardware RAID controller's driver documentation for
information on this.

mdadm

The md ad m command-line tool is used to manage software RAID in Linux, i.e. md rai d . For
information on the different md ad m modes and options, refer to man md ad m. The man page also
contains useful examples for common operations like creating, monitoring, and assembling software
RAID arrays.

dmraid

As the name suggests, d mrai d is used to manage device-mapper RAID sets. The d mrai d tool finds
ATARAID devices using multiple metadata format handlers, each supporting various formats. For a
complete list of supported formats, run d mrai d -l .

As mentioned earlier in Section 17.3, Linux RAID Subsystems , the d mrai d tool cannot configure
RAID sets after creation. For more information about using d mrai d , refer to man d mrai d .

17.7. Advanced RAID Device Creat ion

In some cases, you may wish to install the operating system on an array that can't be created after
the installation completes. Usually, this means setting up the /bo o t or root file system arrays on a
complex RAID device; in such cases, you may need to use array options that are not supported by
An aco n d a. To work around this, perform the following procedure:

Pro ced u re 17.1. Ad van ced R AID d evice creat io n

1. Insert the install disk as you normally would.

2. D uring the initial boot up, select R escu e Mo d e instead of In st all or U p g rad e. When the
system fully boots into Rescue mode, the user will be presented with a command line terminal.

3. From this terminal, use parted to create RAID partitions on the target hard drives. Then, use
md ad m to manually create raid arrays from those partitions using any and all settings and
options available. For more information on how to do these, refer to Chapter 12, Partitions,
man parted , and man md ad m.

4. Once the arrays are created, you can optionally create file systems on the arrays as well.

5. Reboot the computer and this time select In st all or U p g rad e to install as normal. As
An aco n d a searches the disks in the system, it will find the pre-existing RAID devices.

6. When asked about how to use the disks in the system, select C u st o m Layo u t and click
Next. In the device listing, the pre-existing MD RAID devices will be listed.

7. Select a RAID device, click Ed i t and configure its mount point and (optionally) the type of
file system it should use (if you did not create one earlier) then click D o ne. An aco n d a will
perform the install to this pre-existing RAID device, preserving the custom options you
selected when you created it in Rescue Mode.

137
St orage Administ rat ion G uide

Note

The limited Rescue Mode of the installer does not include man pages. Both the man md ad m
and man md contain useful information for creating custom RAID arrays, and may be needed
throughout the workaround. As such, it can be helpful to either have access to a machine with
these man pages present, or to print them out prior to booting into Rescue Mode and creating
your custom arrays.

[2] A ho t-s wap c has s is allo ws yo u to remo ve a hard d rive witho ut having to p o wer-d o wn yo ur s ys tem.

[3] RAID level 1 c o mes at a hig h c o s t b ec aus e yo u write the s ame info rmatio n to all o f the d is ks in the
array, p ro vid es d ata reliab ility, b ut in a muc h les s s p ac e-effic ient manner than p arity b as ed RAID levels
s uc h as level 5. Ho wever, this s p ac e ineffic ienc y c o mes with a p erfo rmanc e b enefit: p arity-b as ed RAID
levels c o ns ume c o ns id erab ly mo re CPU p o wer in o rd er to g enerate the p arity while RAID level 1 s imp ly
writes the s ame d ata mo re than o nc e to the multip le RAID memb ers with very little CPU o verhead . As
s uc h, RAID level 1 c an o utp erfo rm the p arity-b as ed RAID levels o n mac hines where s o ftware RAID is
emp lo yed and CPU res o urc es o n the mac hine are c o ns is tently taxed with o p eratio ns o ther than RAID
ac tivities .

[4] Parity info rmatio n is c alc ulated b as ed o n the c o ntents o f the res t o f the memb er d is ks in the array.
This info rmatio n c an then b e us ed to rec o ns truc t d ata when o ne d is k in the array fails . The
rec o ns truc ted d ata c an then b e us ed to s atis fy I/O req ues ts to the failed d is k b efo re it is rep lac ed and
to rep o p ulate the failed d is k after it has b een rep lac ed .

138
 Chapt er 1 8 . Using t he mount Command

Chapter 18. Using the mo unt Command

On Linux, UNIX, and similar operating systems, file systems on different partitions and removable
devices (CD s, D VD s, or USB flash drives for example) can be attached to a certain point (the mount
point) in the directory tree, and then detached again. To attach or detach a file system, use the mo unt
or umo unt command respectively. This chapter describes the basic use of these commands, as well
as some advanced topics, such as moving a mount point or creating shared subtrees.

18.1. List ing Current ly Mount ed File Syst ems

To display all currently attached file systems, run the mo unt command with no additional arguments:

mo unt

This command displays the list of known mount points. Each line provides important information
about the device name, the file system type, the directory in which it is mounted, and relevant mount
options in the following form:

device on directory type type (options)

The fi nd mnt utility, which allows users to list mounted file systems in a tree-like form, is also
available from Red Hat Enterprise Linux 6.1. To display all currently attached file systems, run the
fi nd mnt command with no additional arguments:

fi nd mnt

18.1.1. Specifying t he File Syst em T ype

By default, the output of the mo unt command includes various virtual file systems such as sysfs
and tmpfs. To display only the devices with a certain file system type, supply the -t option on the
command line:

mo unt -t type

Similarly, to display only the devices with a certain file system type by using the fi nd mnt command,
type:

fi nd mnt -t type

For a list of common file system types, refer to Table 18.1, Common File System Types . For an
example usage, see Example 18.1, Listing Currently Mounted ext4 File Systems .

Examp le 18.1. List in g C u rren t ly Mo u n t ed ext4 File Syst ems

Usually, both / and /bo o t partitions are formatted to use ext4 . To display only the mount points
that use this file system, type the following at a shell prompt:

~]$ mo unt -t ext4

/dev/sda2 on / type ext4 (rw)
/dev/sda1 on /boot type ext4 (rw)

139
St orage Administ rat ion G uide

To list such mount points using the fi nd mnt command, type:

~]$ fi nd mnt -t ext4

TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda2 ext4 rw,realtime,seclabel,barrier=1,data=ordered
/boot /dev/sda1 ext4 rw,realtime,seclabel,barrier=1,data=ordered

18.2. Mount ing a File Syst em

To attach a certain file system, use the mo unt command in the following form:

mo unt [option] device directory

The device can be identified by a full path to a block device (for example, /dev/sda3 ), a universally
unique identifier (UUID ; for example, UUID =34795a28-ca6d-4fd8-a347-73671d0c19cb ), or a volume
label (for example, LABEL=home ). Note that while a file system is mounted, the original content of the
directory is not accessible.

Important

Linux does not prevent a user from mounting a file system to a directory with a file system
already attached to it. To determine whether a particular directory serves as a mount point, run
the fi nd mnt utility with the directory as its argument and verify the exit code:

fi nd mnt directory; echo $?

If no file system is attached to the directory, the above command returns 1.

When you run the mo unt command without all required information, that is without the device name,
the target directory, or the file system type, the mo unt reads the contents of the /etc/fstab file to
check if the given file system is listed. The /etc/fstab file contains a list of device names and the
directories in which the selected file systems are set to be mounted as well as the file system type and
mount options. Therefore, when mounting a file system that is specified in /etc/fstab, you can
choose one of the following options:

mo unt [option] directory

mo unt [option] device

Note that permissions are required to mount the file systems unless the command is run as ro o t (see
Section 18.2.2, Specifying the Mount Options ).

14 0
 Chapt er 1 8 . Using t he mount Command

Note

To determine the UUID andif the device uses itthe label of a particular device, use the
bl ki d command in the following form:

bl ki d device

For example, to display information about /d ev/sd a3, type:

~]# bl ki d /d ev/sd a3
/dev/sda3: LABEL="home" UUID="34795a28-ca6d-4fd8-a347-73671d0c19cb"
TYPE="ext3"

18.2.1. Specifying t he File Syst em T ype

In most cases, mo unt detects the file system automatically. However, there are certain file systems,
such as NFS (Network File System) or C IFS (Common Internet File System), that are not recognized,
and need to be specified manually. To specify the file system type, use the mo unt command in the
following form:

mo unt -t type device directory

Table 18.1, Common File System Types provides a list of common file system types that can be
used with the mo unt command. For a complete list of all available file system types, consult the
relevant manual page as referred to in Section 18.4.1, Manual Page D ocumentation .

T ab le 18.1. C o mmo n File Syst em T yp es

T yp e D escrip t io n
ext2 The ext2 file system.
ext3 The ext3 file system.
ext4 The ext4 file system.
btrfs The btrfs file system.
xfs The xfs file system.
i so 9 6 6 0 The ISO 9 6 6 0 file system. It is commonly used by optical media, typically
CD s.
jfs The JFS file system created by IBM.
nfs The NFS file system. It is commonly used to access files over the network.
nfs4 The NFSv4 file system. It is commonly used to access files over the network.
ntfs The NT FS file system. It is commonly used on machines that are running the
Windows operating system.
ud f The UD F file system. It is commonly used by optical media, typically D VD s.
vfat The FAT file system. It is commonly used on machines that are running the
Windows operating system, and on certain digital media such as USB flash
drives or floppy disks.

See Example 18.2, Mounting a USB Flash D rive for an example usage.

14 1
St orage Administ rat ion G uide

Examp le 18.2. Mo u n t in g a U SB Flash D rive

Older USB flash drives often use the FAT file system. Assuming that such drive uses the
/d ev/sd c1 device and that the /med i a/fl ashd i sk/ directory exists, mount it to this directory
by typing the following at a shell prompt as ro o t:

~]# mo unt -t vfat /d ev/sd c1 /med i a/fl ashd i sk

18.2.2. Specifying t he Mount Opt ions

To specify additional mount options, use the command in the following form:

mo unt -o options device directory

When supplying multiple options, do not insert a space after a comma, or mo unt will incorrectly
interpret the values following spaces as additional parameters.

Table 18.2, Common Mount Options provides a list of common mount options. For a complete list
of all available options, consult the relevant manual page as referred to in Section 18.4.1, Manual
Page D ocumentation .

T ab le 18.2. C o mmo n Mo u n t O p t io n s

O p t io n D escrip t io n
async Allows the asynchronous input/output operations on the file system.
auto Allows the file system to be mounted automatically using the mo unt -a
command.
d efaul ts Provides an alias for async,auto ,d ev,exec,no user,rw,sui d .
exec Allows the execution of binary files on the particular file system.
loop Mounts an image as a loop device.
no auto D efault behavior disallows the automatic mount of the file system using the
mo unt -a command.
no exec D isallows the execution of binary files on the particular file system.
no user D isallows an ordinary user (that is, other than ro o t) to mount and unmount
the file system.
remo unt Remounts the file system in case it is already mounted.
ro Mounts the file system for reading only.
rw Mounts the file system for both reading and writing.
user Allows an ordinary user (that is, other than ro o t) to mount and unmount the
file system.

See Example 18.3, Mounting an ISO Image for an example usage.

Examp le 18.3. Mo u n t in g an ISO Imag e

An ISO image (or a disk image in general) can be mounted by using the loop device. Assuming
that the ISO image of the Fedora 14 installation disc is present in the current working directory and
that the /med i a/cd ro m/ directory exists, mount the image to this directory by running the
following command as ro o t:

14 2
 Chapt er 1 8 . Using t he mount Command

~]# mo unt -o ro ,l o o p Fed o ra-14 -x86 _6 4 -Li ve-D eskto p. i so /med i a/cd ro m

Note that ISO 9660 is by design a read-only file system.

18.2.3. Sharing Mount s

Occasionally, certain system administration tasks require access to the same file system from more
than one place in the directory tree (for example, when preparing a chroot environment). This is
possible, and Linux allows you to mount the same file system to as many directories as necessary.
Additionally, the mo unt command implements the --bi nd option that provides a means for
duplicating certain mounts. Its usage is as follows:

mo unt --bi nd old_directory new_directory

Although this command allows a user to access the file system from both places, it does not apply on
the file systems that are mounted within the original directory. To include these mounts as well, type:

mo unt --rbi nd old_directory new_directory

Additionally, to provide as much flexibility as possible, Red Hat Enterprise Linux 7 implements the
functionality known as shared subtrees. This feature allows the use of the following four mount types:

Sh ared Mo u n t

A shared mount allows the creation of an exact replica of a given mount point. When a
mount point is marked as a shared mount, any mount within the original mount point is
reflected in it, and vice versa. To change the type of a mount point to a shared mount, type
the following at a shell prompt:

mo unt --make-shared mount_point

Alternatively, to change the mount type for the selected mount point and all mount points
under it, type:

mo unt --make-rshared mount_point

See Example 18.4, Creating a Shared Mount Point for an example usage.

Examp le 18.4 . C reat in g a Sh ared Mo u n t Po in t

There are two places where other file systems are commonly mounted: the /med i a
directory for removable media, and the /mnt directory for temporarily mounted file
systems. By using a shared mount, you can make these two directories share the same
content. To do so, as ro o t, mark the /med i a directory as shared :

~]# mo unt --bi nd /med i a /med i a

~]# mo unt --make-shared /med i a

Then create its duplicate in /mnt by using the following command:

~]# mo unt --bi nd /med i a /mnt

14 3
St orage Administ rat ion G uide

It is now possible to verify that a mount within /med i a also appears in /mnt. For
example, if the CD -ROM drive contains non-empty media and the /med i a/cd ro m/
directory exists, run the following commands:

~]# mo unt /d ev/cd ro m /med i a/cd ro m

~]# l s /med i a/cd ro m
EFI GPL isolinux LiveOS
~]# l s /mnt/cd ro m
EFI GPL isolinux LiveOS

Similarly, it is possible to verify that any file system mounted in the /mnt directory is
reflected in /med i a. For instance, if a non-empty USB flash drive that uses the
/d ev/sd c1 device is plugged in and the /mnt/fl ashd i sk/ directory is present, type:

~]# mo unt /d ev/sd c1 /mnt/fl ashd i sk

~]# l s /med i a/fl ashd i sk
en-US publican.cfg
~]# l s /mnt/fl ashd i sk
en-US publican.cfg

Slave Mo u n t

A slave mount allows the creation of a limited duplicate of a given mount point. When a
mount point is marked as a slave mount, any mount within the original mount point is
reflected in it, but no mount within a slave mount is reflected in its original. To change the
type of a mount point to a slave mount, type the following at a shell prompt:

mo unt --make-sl ave mount_point

Alternatively, it is possible to change the mount type for the selected mount point and all
mount points under it by typing:

mo unt --make-rsl ave mount_point

See Example 18.5, Creating a Slave Mount Point for an example usage.

Examp le 18.5. C reat in g a Slave Mo u n t Po in t

This example shows how to get the content of the /med i a directory to appear in /mnt as
well, but without any mounts in the /mnt directory to be reflected in /med i a. As ro o t,
first mark the /med i a directory as shared :

~]# mo unt --bi nd /med i a /med i a

~]# mo unt --make-shared /med i a

Then create its duplicate in /mnt, but mark it as slave :

~]# mo unt --bi nd /med i a /mnt

~]# mo unt --make-sl ave /mnt

Now verify that a mount within /med i a also appears in /mnt. For example, if the CD -
ROM drive contains non-empty media and the /med i a/cd ro m/ directory exists, run the
following commands:

14 4
 Chapt er 1 8 . Using t he mount Command

~]# mo unt /d ev/cd ro m /med i a/cd ro m

~]# l s /med i a/cd ro m
EFI GPL isolinux LiveOS
~]# l s /mnt/cd ro m
EFI GPL isolinux LiveOS

Also verify that file systems mounted in the /mnt directory are not reflected in /med i a.
For instance, if a non-empty USB flash drive that uses the /d ev/sd c1 device is plugged
in and the /mnt/fl ashd i sk/ directory is present, type:

~]# mo unt /d ev/sd c1 /mnt/fl ashd i sk

~]# l s /med i a/fl ashd i sk
~]# l s /mnt/fl ashd i sk
en-US publican.cfg

Privat e Mo u n t

A private mount is the default type of mount, and unlike a shared or slave mount, it does not
receive or forward any propagation events. To explicitly mark a mount point as a private
mount, type the following at a shell prompt:

mo unt --make-pri vate mount_point

Alternatively, it is possible to change the mount type for the selected mount point and all
mount points under it:

mo unt --make-rpri vate mount_point

See Example 18.6, Creating a Private Mount Point for an example usage.

Examp le 18.6 . C reat in g a Privat e Mo u n t Po in t

Taking into account the scenario in Example 18.4, Creating a Shared Mount Point ,
assume that a shared mount point has been previously created by using the following
commands as ro o t:

~]# mo unt --bi nd /med i a /med i a

~]# mo unt --make-shared /med i a
~]# mo unt --bi nd /med i a /mnt

To mark the /mnt directory as private , type:

~]# mo unt --make-pri vate /mnt

It is now possible to verify that none of the mounts within /med i a appears in /mnt. For
example, if the CD -ROM drives contains non-empty media and the /med i a/cd ro m/
directory exists, run the following commands:

~]# mo unt /d ev/cd ro m /med i a/cd ro m

~]# l s /med i a/cd ro m
EFI GPL isolinux LiveOS
~]# l s /mnt/cd ro m
~]#

14 5
St orage Administ rat ion G uide

It is also possible to verify that file systems mounted in the /mnt directory are not
reflected in /med i a. For instance, if a non-empty USB flash drive that uses the
/d ev/sd c1 device is plugged in and the /mnt/fl ashd i sk/ directory is present, type:

~]# mo unt /d ev/sd c1 /mnt/fl ashd i sk

~]# l s /med i a/fl ashd i sk
~]# l s /mnt/fl ashd i sk
en-US publican.cfg

U n b in d ab le Mo u n t

In order to prevent a given mount point from being duplicated whatsoever, an unbindable
mount is used. To change the type of a mount point to an unbindable mount, type the
following at a shell prompt:

mo unt --make-unbi nd abl e mount_point

Alternatively, it is possible to change the mount type for the selected mount point and all
mount points under it:

mo unt --make-runbi nd abl e mount_point

See Example 18.7, Creating an Unbindable Mount Point for an example usage.

Examp le 18.7. C reat in g an U n b in d ab le Mo u n t Po in t

To prevent the /med i a directory from being shared, as ro o t, type the following at a
shell prompt:

~]# mo unt --bi nd /med i a /med i a

~]# mo unt --make-unbi nd abl e /med i a

This way, any subsequent attempt to make a duplicate of this mount will fail with an error:

~]# mo unt --bi nd /med i a /mnt

mount: wrong fs type, bad option, bad superblock on /media,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

18.2.4 . Moving a Mount Point

To change the directory in which a file system is mounted, use the following command:

mo unt --mo ve old_directory new_directory

See Example 18.8, Moving an Existing NFS Mount Point for an example usage.

Examp le 18.8. Mo vin g an Exist in g N FS Mo u n t Po in t

14 6
 Chapt er 1 8 . Using t he mount Command

An NFS storage contains user directories and is already mounted in /mnt/userd i rs/. As ro o t,
move this mount point to /ho me by using the following command:

~]# mo unt --mo ve /mnt/userd i rs /ho me

To verify the mount point has been moved, list the content of both directories:

~]# l s /mnt/userd i rs
~]# l s /ho me
jill joe

18.2.5. Set t ing Read-only Permissions for ro o t

Sometimes, you need to mount the root file system with read-only permissions. Example use cases
include enhancing security or ensuring data integrity after an unexpected system power-off.

1 8 .2 .5 .1 . Co nfiguring ro o t t o Mo unt wit h Re ad-o nly Pe rm issio ns o n Bo o t

1. In the /etc/sysco nfi g /read o nl y-ro o t file, change R EAD O NLY to yes:

# Set to 'yes' to mount the system file systems read-only.

READONLY=yes
[output truncated]

2. Change d efaul ts to ro in the root entry (/) in the /etc/fstab file:

/dev/mapper/luks-c376919e... / ext4 ro ,x-systemd.device-timeout=0 1

1

3. Add ro to the G R UB_C MD LINE_LINUX directive in the /etc/d efaul t/g rub file and ensure
that it does not contain rw:

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root
rd.lvm.lv=rhel/swap rhgb quiet ro "

4. Recreate the GRUB2 configuration file:

~]# g rub2-mkco nfi g -o /bo o t/g rub2/g rub. cfg

5. If you need to add files and directories to be mounted with write permissions in the tmpfs file
system, create a text file in the /etc/rwtab. d / directory and put the configuration there. For
example, to mount /etc/exampl e/fi l e with write permissions, add this line to the
/etc/rwtab. d /example file:

files /etc/example/file

Important

Changes made to files and directories in tmpfs do not persist across boots.

14 7
St orage Administ rat ion G uide

See Section 18.2.5.3, Files and D irectories That Retain Write Permissions for more
information on this step.

6. Reboot the system.

1 8 .2 .5 .2 . Re m o unt ing ro o t Inst ant ly

If root (/) was mounted with read-only permissions on system boot, you can remount it with write
permissions:

~]# mo unt -o remo unt,rw /

This can be particularly useful when / is incorrectly mounted with read-only permissions.

To remount / with read-only permissions again, run:

~]# mo unt -o remo unt,ro /

Note

This command mounts the whole / with read-only permissions. A better approach is to retain
write permissions for certain files and directories by copying them into RAM, as described in
Section 18.2.5.1, Configuring ro o t to Mount with Read-only Permissions on Boot .

1 8 .2 .5 .3. File s and Dire ct o rie s T hat Re t ain Writ e Pe rm issio ns

For the system to function properly, some files and directories need to retain write permissions. With
root in read-only mode, they are mounted in RAM in the tmpfs temporary file system. The default set
of such files and directories is read from the /etc/rwtab file, which contains:

dirs /var/cache/man
dirs /var/gdm
[output truncated]
empty /tmp
empty /var/cache/foomatic
[output truncated]
files /etc/adjtime
files /etc/ntp.conf
[output truncated]

Entries in the /etc/rwtab file follow this format:

how the file or directory is copied to tmpfs path to the file or

directory

A file or directory can be copied to tmpfs in three ways, so there are three types of entries:

empty path: An empty path is copied to tmpfs. Example: empty /tmp

d i rs path: A directory tree is copied to tmpfs, empty. Example: d i rs /var/run

fi l es path: A file or a directory tree is copied to tmpfs intact. Example: fi l es

/etc/reso l v. co nf

14 8
 Chapt er 1 8 . Using t he mount Command

The same format applies when adding custom paths to /etc/rwtab. d /.

18.3. Unmount ing a File Syst em

To detach a previously mounted file system, use either of the following variants of the umo unt
command:

umo unt directory

umo unt device

Note that unless this is performed while logged in as ro o t, the correct permissions must be available
to unmount the file system (see Section 18.2.2, Specifying the Mount Options ). See Example 18.9,
Unmounting a CD for an example usage.

Important

When a file system is in use (for example, when a process is reading a file on this file system,
or when it is used by the kernel), running the umo unt command will fail with an error. To
determine which processes are accessing the file system, use the fuser command in the
following form:

fuser -m directory

For example, to list the processes that are accessing a file system mounted to the
/med i a/cd ro m/ directory, type:

~]$ fuser -m /med i a/cd ro m

/media/cdrom: 1793 2013 2022 2435 10532c 10672c

Examp le 18.9 . U n mo u n t in g a C D

To unmount a CD that was previously mounted to the /med i a/cd ro m/ directory, type the
following at a shell prompt:

~]$ umo unt /med i a/cd ro m

18.4 . mo unt Command References

The following resources provide an in-depth documentation on the subject.

18.4 .1. Manual Page Document at ion

man 8 mo unt The manual page for the mo unt command that provides a full documentation
on its usage.

man 8 umo unt The manual page for the umo unt command that provides a full documentation
on its usage.

14 9
St orage Administ rat ion G uide

man 8 fi nd mnt The manual page for the fi nd mnt command that provides a full
documentation on its usage.

man 5 fstab The manual page providing a thorough description of the /etc/fstab file
format.

18.4 .2. Useful Websit es

Shared subtrees An LWN article covering the concept of shared subtrees.

150
 Chapt er 1 9 . T he volume_key Funct ion

Chapter 19. The vo l ume_key Function

The volume_key function provides two tools, libvolume_key and vo l ume_key. libvolume_key is a
library for manipulating storage volume encryption keys and storing them separately from volumes.
vo l ume_key is an associated command line tool used to extract keys and passphrases in order to
restore access to an encrypted hard drive.

This is useful for when the primary user forgets their keys and passwords, after an employee leaves
abruptly, or in order to extract data after a hardware or software failure corrupts the header of the
encrypted volume. In a corporate setting, the IT help desk can use vo l ume_key to back up the
encryption keys before handing over the computer to the end user.

Currently, vo l ume_key only supports the LUKS volume encryption format.

Note

vo l ume_key is not included in a standard install of Red Hat Enterprise Linux 7 server. For
information on installing it, refer to
http://fedoraproject.org/wiki/D isk_encryption_key_escrow_use_cases.

19.1. vo l ume_key Commands

The format for vo l ume_key is:

vo l ume_key [O P T IO N]. . . O P ER AND

The operands and mode of operation for vo l ume_key are determined by specifying one of the
following options:

--save

This command expects the operand volume [packet]. If a packet is provided then
vo l ume_key will extract the keys and passphrases from it. If packet is not provided, then
vo l ume_key will extract the keys and passphrases from the volume, prompting the user
where necessary. These keys and passphrases will then be stored in one or more output
packets.

--resto re

This command expects the operands volume packet. It then opens the volume and uses the
keys and passphrases in the packet to make the volume accessible again, prompting the
user where necessary, such as allowing the user to enter a new passphrase, for example.

--setup-vo l ume

This command expects the operands volume packet name. It then opens the volume and uses
the keys and passphrases in the packet to set up the volume for use of the decrypted data as
name.

Name is the name of a dm-crypt volume. This operation makes the decrypted volume
available as /d ev/mapper/name.

This operation does not permanently alter the volume by adding a new passphrase, for
example. The user can access and modify the decrypted volume, modifying volume in the

151
St orage Administ rat ion G uide

process.

--reencrypt, --secrets, an d --d ump

These three commands perform similar functions with varying output methods. They each
require the operand packet, and each opens the packet, decrypting it where necessary. --
reencrypt then stores the information in one or more new output packets. --secrets
outputs the keys and passphrases contained in the packet. --d ump outputs the content of
the packet, though the keys and passphrases are not output by default. This can be
changed by appending --wi th-secrets to the command. It is also possible to only dump
the unencrypted parts of the packet, if any, by using the --unencrypted command. This
does not require any passphrase or private key access.

Each of these can be appended with the following options:

-o , --o utput packet

This command writes the default key or passphrase to the packet. The default key or
passphrase depends on the volume format. Ensure it is one that is unlikely to expire, and
will allow --resto re to restore access to the volume.

--o utput-fo rmat format

This command uses the specified format for all output packets. Currently, format can be one
of the following:

asymmetri c: uses CMS to encrypt the whole packet, and requires a certificate

asymmetri c_wrap_secret_o nl y: wraps only the secret, or keys and passphrases,

and requires a certificate

passphrase: uses GPG to encrypt the whole packet, and requires a passphrase

--create-rand o m-passphrase packet

This command generates a random alphanumeric passphrase, adds it to the volume

(without affecting other passphrases), and then stores this random passphrase into the
packet.

19.2. Using vo l ume_key as an individual user

As an individual user, vo l ume_key can be used to save encryption keys by using the following
procedure.

Note

For all examples in this file, /path/to/volume is a LUKS device, not the plaintext device
contained within. bl ki d -s type /path/to/volume should report
type= "crypto _LUKS".

Pro ced u re 19 .1. U sin g vo l ume_key st an d - alo n e

1. Run:

vo l ume_key --save /path/to/volume -o escro w-packet

152
 Chapt er 1 9 . T he volume_key Funct ion

A prompt will then appear requiring an escrow packet passphrase to protect the key.

2. Save the generated escro w-packet file, ensuring that the passphrase is not forgotten.

If the volume passphrase is forgotten, use the saved escrow packet to restore access to the data.

Pro ced u re 19 .2. R est o re access t o d at a wit h escro w p acket

1. Boot the system in an environment where vo l ume_key can be run and the escrow packet is
available (a rescue mode, for example).

2. Run:

vo l ume_key --resto re /path/to/volume escro w-packet

A prompt will appear for the escrow packet passphrase that was used when creating the
escrow packet, and for the new passphrase for the volume.

3. Mount the volume using the chosen passphrase.

To free up the passphrase slot in the LUKS header of the encrypted volume, remove the old, forgotten
passphrase by using the command cryptsetup l uksKi l l Sl o t.

19.3. Using vo l ume_key in a larger organiz at ion

In a larger organization, using a single password known by every system administrator and keeping
track of a separate password for each system is impractical and a security risk. To counter this,
vo l ume_key can use asymmetric cryptography to minimize the number of people who know the
password required to access encrypted data on any computer.

This section will cover the procedures required for preparation before saving encryption keys, how to
save encryption keys, restoring access to a volume, and setting up emergency passphrases.

19.3.1. Preparat ion for saving encrypt ion keys

In order to begin saving encryption keys, some preparation is required.

Pro ced u re 19 .3. Prep arat io n

1. Create an X509 certificate/private pair.

2. D esignate trusted users who are trusted not to compromise the private key. These users will
be able to decrypt the escrow packets.

3. Choose which systems will be used to decrypt the escrow packets. On these systems, set up
an NSS database that contains the private key.

If the private key was not created in an NSS database, follow these steps:

A. Store the certificate and private key in an P KC S#12 file.

B. Run:

certuti l -d /the/nss/directory -N

153
St orage Administ rat ion G uide

At this point it is possible to choose an NSS database password. Each NSS database can
have a different password so the designated users do not need to share a single
password if a separate NSS database is used by each user.

C. Run:

pk12uti l -d /the/nss/directory -i the-pkcs12-file

4. D istribute the certificate to anyone installing systems or saving keys on existing systems.

5. For saved private keys, prepare storage that allows them to be looked up by machine and
volume. For example, this can be a simple directory with one subdirectory per machine, or a
database used for other system management tasks as well.

19.3.2. Saving encrypt ion keys

After completing the required preparation (see Section 19.3.1, Preparation for saving encryption
keys ) it is now possible to save the encryption keys using the following procedure.

Note

For all examples in this file, /path/to /vo l ume is a LUKS device, not the plaintext device
contained within; bl ki d -s type /path/to/volume should report
type= "crypto _LUKS".

Pro ced u re 19 .4 . Savin g en cryp t io n keys

1. Run:

vo l ume_key --save /path/to/volume -c /path/to/cert escro w-packet

2. Save the generated escro w-packet file in the prepared storage, associating it with the
system and the volume.

These steps can be performed manually, or scripted as part of system installation.

19.3.3. Rest oring access t o a volume

After the encryption keys have been saved (see Section 19.3.1, Preparation for saving encryption
keys and Section 19.3.2, Saving encryption keys ), access can be restored to a driver where
needed.

Pro ced u re 19 .5. R est o rin g access t o a vo lu me

1. Get the escrow packet for the volume from the packet storage and send it to one of the
designated users for decryption.

2. The designated user runs:

vo l ume_key --reencrypt -d /the/nss/directory escro w-packet-i n -o

escro w-packet-o ut

154
 Chapt er 1 9 . T he volume_key Funct ion

After providing the NSS database password, the designated user chooses a passphrase for
encrypting escro w-packet-o ut. This passphrase can be different every time and only
protects the encryption keys while they are moved from the designated user to the target
system.

3. Obtain the escro w-packet-o ut file and the passphrase from the designated user.

4. Boot the target system in an environment that can run vo l ume_key and have the escro w-
packet-o ut file available, such as in a rescue mode.

5. Run:

vo l ume_key --resto re /path/to/volume escro w-packet-o ut

A prompt will appear for the packet passphrase chosen by the designated user, and for a new
passphrase for the volume.

6. Mount the volume using the chosen volume passphrase.

It is possible to remove the old passphrase that was forgotten by using cryptsetup
l uksKi l l Sl o t, for example, to free up the passphrase slot in the LUKS header of the encrypted
volume. This is done with the command cryptsetup l uksKi l l Sl o t device key-slot. For
more information and examples see cryptsetup --hel p.

19.3.4 . Set t ing up emergency passphrases

In some circumstances (such as traveling for business) it is impractical for system administrators to
work directly with the affected systems, but users still need access to their data. In this case,
vo l ume_key can work with passphrases as well as encryption keys.

D uring the system installation, run:

vo l ume_key --save /path/to/volume -c /path/to/ert --create-rand o m-

passphrase passphrase-packet

This generates a random passphrase, adds it to the specified volume, and stores it to passphrase-
packet. It is also possible to combine the --create-rand o m-passphrase and -o options to
generate both packets at the same time.

If a user forgets the password, the designated user runs:

vo l ume_key --secrets -d /your/nss/directory passphrase-packet

This shows the random passphrase. Give this passphrase to the end user.

19.4 . vo l ume_key References

More information on vo l ume_key can be found:

in the readme file located at /usr/share/d o c/vo l ume_key-*/R EAD ME

on vo l ume_key's manpage using man vo l ume_key

online at http://fedoraproject.org/wiki/D isk_encryption_key_escrow_use_cases

155
St orage Administ rat ion G uide

Chapter 20. Solid-State Disk Deployment Guidelines

Solid-state disks (SSD ) are storage devices that use NAND flash chips to persistently store data. This
sets them apart from previous generations of disks, which store data in rotating, magnetic platters. In
an SSD , the access time for data across the full Logical Block Address (LBA) range is constant;
whereas with older disks that use rotating media, access patterns that span large address ranges
incur seek costs. As such, SSD devices have better latency and throughput.

Performance degrades as the number of used blocks approaches the disk capacity. The degree of
performance impact varies greatly by vendor. However, all devices experience some degradation.

To address the degradation issue, the host system (for example, the Linux kernel) may use discard
requests to inform the storage that a given range of blocks is no longer in use. An SSD can use this
information to free up space internally, using the free blocks for wear-leveling. D iscards will only be
issued if the storage advertises support in terms of its storage protocol (be it ATA or SCSI). D iscard
requests are issued to the storage using the negotiated discard command specific to the storage
protocol (T R IM command for ATA, and WR IT E SAME with UNMAP set, or UNMAP command for SCSI).

Enabling d i scard support is most useful when the following two points are true:

Free space is still available on the file system.

Most logical blocks on the underlying storage device have already been written to.

For more information about T R IM, refer to its Data Set Management T13 Specifications from the
following link:

http://t13.org/D ocuments/UploadedD ocuments/docs2008/e07154r6-

D ata_Set_Management_Proposal_for_ATA-ACS2.doc

For more information about UNMAP , refer to section 4.7.3.4 of the SCSI Block Commands 3 T10
Specification from the following link:

http://www.t10.org/cgi-bin/ac.pl?t=f&f=sbc3r26.pdf

Note

Not all solid-state devices in the market have d i scard support. To determine if your solid-
state device has d i scard support check for
/sys/bl o ck/sd a/q ueue/d i scard _g ranul ari ty.

Deployment Considerat ions

Because of the internal layout and operation of SSD s, it is best to partition devices on an internal
erase block boundary. Partitioning utilities in Red Hat Enterprise Linux 7 chooses sane defaults if the
SSD exports topology information.

However, if the device does not export topology information, Red Hat recommends that the first
partition be created at a 1MB boundary.

The Logical Volume Manager (LVM), the device-mapper (D M) targets, and MD (software raid) targets
that LVM uses support discards. The only D M targets that do not support discards are dm-snapshot,
dm-crypt, and dm-raid45. D iscard support for the dm-mirror was added in Red Hat
Enterprise Linux 6.1 and as of 7.0 MD supports discards.

156
 Chapt er 2 0 . Solid- St at e Disk Deployment G uidelines

Using RAID level 5 over SSD results in low performance if SSD s do not handle d i scard correctly.
You can set discard in the rai d 4 56 . co nf file, or in the GRUB2 configuration. For instructions, see
the following procedures.

Pro ced u re 20.1. Set t in g d iscard in raid 4 56 .co n f

The devices_handle_discard_safely module parameter is set in the rai d 4 56 module. To

enable discard in the rai d 4 56 . co nf file:

1. Verify that your hardware supports discards:

# cat /sys/bl o ck/disk-name/q ueue/d i scard _zero es_d ata

If the returned value is 1, discards are supported. If the command returns 0 , the RAID code
has to zero the disk out, which takes more time.

2. Create the /etc/mo d pro be. d /rai d 4 56 . co nf file, and include the following line:

options raid456 devices_handle_discard_safely=Y

3. Use the d racut -f command to rebuild the initial ramdisk (i ni trd ).

4. Reboot the system for the changes to take effect.

Pro ced u re 20.2. Set t in g d iscard in t h e G R U B 2 C o n f ig u rat io n

The devices_handle_discard_safely module parameter is set in the rai d 4 56 module. To

enable discard in the GRUB2 configuration:

1. Verify that your hardware supports discards:

# cat /sys/bl o ck/disk-name/q ueue/d i scard _zero es_d ata

If the returned value is 1, discards are supported. If the command returns 0 , the RAID code
has to zero the disk out, which takes more time.

2. Add the following line to the /etc/d efaul t/g rub file:

raid456.devices_handle_discard_safely=Y

3. The location of the GRUB2 configuration file is different on systems with the BIOS firmware
and on systems with UEFI. Use one of the following commands to recreate the GRUB2
configuration file.

On a system with the BIOS firmware, use:

# g rub2-mkco nfi g -o /bo o t/g rub2/g rub. cfg

On a system with the UEFI firmware, use:

# g rub2-mkco nfi g -o /bo o t/efi /EFI/red hat/g rub. cfg

4. Reboot the system for the changes to take effect.

157
St orage Administ rat ion G uide

Note

In Red Hat Enterprise Linux 7, only the ext4 and XFS file systems fully support discard.

In Red Hat Enterprise Linux 6.3 and earlier, only the ext4 file system fully supports discard. Starting
with Red Hat Enterprise Linux 6.4, both ext4 and XFS file systems fully support discard. To enable
discard commands on a device, use the d i scard option of the mo unt command. For example, to
mount /d ev/sd a2 to /mnt with discard enabled, use:

# mo unt -t ext4 -o d i scard /d ev/sd a2 /mnt

By default, ext4 does not issue the d i scard command to, primarily, avoid problems on devices
which might not properly implement discard. The Linux swap code issues d i scard commands to
discard-enabled devices, and there is no option to control this behavior.

Performance T uning Considerat ions

For information on performance tuning considerations regarding solid-state disks, see the Solid-
State D isks section in the Red Hat Enterprise Linux 7 Performance Tuning Guide.

158
 Chapt er 2 1 . Writ e Barriers

Chapter 21. Write Barriers

A write barrier is a kernel mechanism used to ensure that file system metadata is correctly written and
ordered on persistent storage, even when storage devices with volatile write caches lose power. File
systems with write barriers enabled also ensure that data transmitted via fsync() is persistent
throughout a power loss.

Enabling write barriers incurs a substantial performance penalty for some applications. Specifically,
applications that use fsync() heavily or create and delete many small files will likely run much
slower.

21.1. Import ance of Writ e Barriers

File systems take great care to safely update metadata, ensuring consistency. Journalled file systems
bundle metadata updates into transactions and send them to persistent storage in the following
manner:

1. First, the file system sends the body of the transaction to the storage device.

2. Then, the file system sends a commit block.

3. If the transaction and its corresponding commit block are written to disk, the file system
assumes that the transaction will survive any power failure.

However, file system integrity during power failure becomes more complex for storage devices with
extra caches. Storage target devices like local S-ATA or SAS drives may have write caches ranging
from 32MB to 64MB in size (with modern drives). Hardware RAID controllers often contain internal
write caches. Further, high end arrays, like those from NetApp, IBM, Hitachi and EMC (among others),
also have large caches.

Storage devices with write caches report I/O as " complete" when the data is in cache; if the cache
loses power, it loses its data as well. Worse, as the cache de-stages to persistent storage, it may
change the original metadata ordering. When this occurs, the commit block may be present on disk
without having the complete, associated transaction in place. As a result, the journal may replay
these uninitialized transaction blocks into the file system during post-power-loss recovery; this will
cause data inconsistency and corruption.

How Writ e Barriers Work

Write barriers are implemented in the Linux kernel via storage write cache flushes before and after the
I/O, which is order-critical. After the transaction is written, the storage cache is flushed, the commit
block is written, and the cache is flushed again. This ensures that:

The disk contains all the data.

No re-ordering has occurred.

With barriers enabled, an fsync() call will also issue a storage cache flush. This guarantees that
file data is persistent on disk even if power loss occurs shortly after fsync() returns.

21.2. Enabling/Disabling Writ e Barriers

159
St orage Administ rat ion G uide

To mitigate the risk of data corruption during power loss, some storage devices use battery-backed
write caches. Generally, high-end arrays and some hardware controllers use battery-backed write
caches. However, because the cache's volatility is not visible to the kernel, Red Hat Enterprise Linux 7
enables write barriers by default on all supported journaling file systems.

Note

Write caches are designed to increase I/O performance. However, enabling write barriers
means constantly flushing these caches, which can significantly reduce performance.

For devices with non-volatile, battery-backed write caches and those with write-caching disabled,
you can safely disable write barriers at mount time using the -o no barri er option for mo unt.
However, some devices do not support write barriers; such devices will log an error message to
/var/l o g /messag es (refer to Table 21.1, Write barrier error messages per file system ).

T ab le 21.1. Writ e b arrier erro r messag es p er f ile syst em

File Syst em Erro r Messag e

ext3/ext4 JBD : barri er-based sync fai l ed o n
device - d i sabl i ng barri ers
XFS Fi l esystem device - D i sabl i ng
barri ers, tri al barri er wri te fai l ed
btrfs btrfs: d i sabl i ng barri ers o n d ev
device

21.3. Writ e Barrier Considerat ions

Some system configurations do not need write barriers to protect data. In most cases, other methods
are preferable to write barriers, since enabling write barriers causes a significant performance
penalty.

Disabling Writ e Caches

One way to alternatively avoid data integrity issues is to ensure that no write caches lose data on
power failures. When possible, the best way to configure this is to simply disable the write cache. On
a simple server or desktop with one or more SATA drives (off a local SATA controller Intel AHCI part),
you can disable the write cache on the target SATA drives with the hd parm command, as in:

# hdparm -W0 /device/

Bat t ery-Backed Writ e Caches

Write barriers are also unnecessary whenever the system uses hardware RAID controllers with
battery-backed write cache. If the system is equipped with such controllers and if its component
drives have write caches disabled, the controller will advertise itself as a write-through cache; this will
inform the kernel that the write cache data will survive a power loss.

Most controllers use vendor-specific tools to query and manipulate target drives. For example, the
LSI Megaraid SAS controller uses a battery-backed write cache; this type of controller requires the
Meg aC l i 6 4 tool to manage target drives. To show the state of all back-end drives for LSI Megaraid
SAS, use:

160
 Chapt er 2 1 . Writ e Barriers

# MegaCli64 -LDGetProp -DskCache -LAll -aALL

To disable the write cache of all back-end drives for LSI Megaraid SAS, use:

# MegaCli64 -LDSetProp -DisDskCache -Lall -aALL

Note

Hardware RAID cards recharge their batteries while the system is operational. If a system is
powered off for an extended period of time, the batteries will lose their charge, leaving stored
data vulnerable during a power failure.

High-End Arrays

High-end arrays have various ways of protecting data in the event of a power failure. As such, there
is no need to verify the state of the internal drives in external RAID storage.

NFS

NFS clients do not need to enable write barriers, since data integrity is handled by the NFS server
side. As such, NFS servers should be configured to ensure data persistence throughout a power loss
(whether through write barriers or other means).

161
St orage Administ rat ion G uide

Chapter 22. Storage I/O Alignment and Size

Recent enhancements to the SCSI and ATA standards allow storage devices to indicate their
preferred (and in some cases, required) I/O alignment and I/O size. This information is particularly
useful with newer disk drives that increase the physical sector size from 512 bytes to 4k bytes. This
information may also be beneficial for RAID devices, where the chunk size and stripe size may impact
performance.

The Linux I/O stack has been enhanced to process vendor-provided I/O alignment and I/O size
information, allowing storage management tools (parted , l vm, mkfs. *, and the like) to optimize
data placement and access. If a legacy device does not export I/O alignment and size data, then
storage management tools in Red Hat Enterprise Linux 7 will conservatively align I/O on a 4k (or
larger power of 2) boundary. This will ensure that 4k-sector devices operate correctly even if they do
not indicate any required/preferred I/O alignment and size.

Refer to Section 22.2, Userspace Access to learn how to determine the information that the
operating system obtained from the device. This data is subsequently used by the storage
management tools to determine data placement.

The IO scheduler has changed for Red Hat Enterprise Linux 7. D efault IO Scheduler is now Deadline,
except for SATA drives. CFQ is the default IO scheduler for SATA drives. For faster storage, D eadline
outperforms CFQ and when it is used there is a performance increase without the need of special
tuning.

If default is not right for some disks (for example, SAS rotational disks), then change the IO scheduler
to CFQ. This instance will depend on the workload.

22.1. Paramet ers for St orage Access

The operating system uses the following information to determine I/O alignment and size:

p h ysical_b lo ck_siz e

Smallest internal unit on which the device can operate

lo g ical_b lo ck_siz e

Used externally to address a location on the device

alig n men t _o f f set

The number of bytes that the beginning of the Linux block device (partition/MD /LVM device)
is offset from the underlying physical alignment

min imu m_io _siz e

The devices preferred minimum unit for random I/O

o p t imal_io _siz e

The devices preferred unit for streaming I/O

For example, certain 4K sector devices may use a 4K physi cal _bl o ck_si ze internally but expose
a more granular 512-byte l o g i cal _bl o ck_si ze to Linux. This discrepancy introduces potential
for misaligned I/O. To address this, the Red Hat Enterprise Linux 7 I/O stack will attempt to start all
data areas on a naturally-aligned boundary (physi cal _bl o ck_si ze) by making sure it accounts
for any alignment_offset if the beginning of the block device is offset from the underlying physical
alignment.

162
 Chapt er 2 2 . St orage I/O Alignment and Siz e

Storage vendors can also supply I/O hints about the preferred minimum unit for random I/O
(mi ni mum_i o _si ze) and streaming I/O (o pti mal _i o _si ze) of a device. For example,
mi ni mum_i o _si ze and o pti mal _i o _si ze may correspond to a RAID device's chunk size and
stripe size respectively.

22.2. Userspace Access

Always take care to use properly aligned and sized I/O. This is especially important for D irect I/O
access. D irect I/O should be aligned on a l o g i cal _bl o ck_si ze boundary, and in multiples of
the l o g i cal _bl o ck_si ze.

With native 4K devices (i.e. l o g i cal _bl o ck_si ze is 4K) it is now critical that applications perform
direct I/O in multiples of the device's l o g i cal _bl o ck_si ze. This means that applications will fail
with native 4k devices that perform 512-byte aligned I/O rather than 4k-aligned I/O.

To avoid this, an application should consult the I/O parameters of a device to ensure it is using the
proper I/O alignment and size. As mentioned earlier, I/O parameters are exposed through the both
sysfs and block device i o ctl interfaces.

For more details, refer to man l i bbl ki d . This man page is provided by the l i bbl ki d -d evel
package.

sysfs Int erface

/sys/block/disk/alignment_offset

or

/sys/block/disk/partition/alignment_offset

Note

The file location depends on whether the disk is a physical disk (be that a local disk, local
RAID , or a multipath LUN) or a virutal disk. The first file location is applicable to physical
disks while the second file location is applicable to virtual disks. The reason for this is
because virtio-blk will always report an alignment value for the partition. Physical disks
may or may not report an alignment value.

/sys/block/disk/queue/physical_block_size

/sys/block/disk/queue/logical_block_size

/sys/block/disk/queue/minimum_io_size

/sys/block/disk/queue/optimal_io_size

The kernel will still export these sysfs attributes for " legacy" devices that do not provide I/O
parameters information, for example:

Examp le 22.1. sysfs in t erf ace

alignment_offset: 0
physical_block_size: 512

163
St orage Administ rat ion G uide

logical_block_size: 512
minimum_io_size: 512
optimal_io_size: 0

Block Device ioct ls

BLKALIG NO FF: al i g nment_o ffset

BLKP BSZG ET : physi cal _bl o ck_si ze

BLKSSZG ET : l o g i cal _bl o ck_si ze

BLKIO MIN: mi ni mum_i o _si ze

BLKIO O P T : o pti mal _i o _si ze

22.3. St andards
This section describes I/O standards used by ATA and SCSI devices.

AT A

ATA devices must report appropriate information via the ID ENT IFY D EVIC E command. ATA devices
only report I/O parameters for physi cal _bl o ck_si ze, l o g i cal _bl o ck_si ze, and
al i g nment_o ffset. The additional I/O hints are outside the scope of the ATA Command Set.

SCSI

I/O parameters support in Red Hat Enterprise Linux 7 requires at least version 3 of the SCSI Primary
Commands (SPC-3) protocol. The kernel will only send an extended inquiry (which gains access to the
BLO C K LIMIT S VP D page) and R EAD C AP AC IT Y (16 ) command to devices which claim
compliance with SPC-3.

The R EAD C AP AC IT Y (16 ) command provides the block sizes and alignment offset:

LO G IC AL BLO C K LENG T H IN BY T ES is used to derive

/sys/bl o ck/disk/q ueue/physi cal _bl o ck_si ze

LO G IC AL BLO C KS P ER P HY SIC AL BLO C K EXP O NENT is used to derive

/sys/bl o ck/disk/q ueue/l o g i cal _bl o ck_si ze

LO WEST ALIG NED LO G IC AL BLO C K AD D R ESS is used to derive:

/sys/bl o ck/disk/al i g nment_o ffset

/sys/bl o ck/disk/partition/al i g nment_o ffset

The BLO C K LIMIT S VP D page (0 xb0 ) provides the I/O hints. It also uses O P T IMAL T R ANSFER
LENG T H G R ANULAR IT Y and O P T IMAL T R ANSFER LENG T H to derive:

/sys/bl o ck/disk/q ueue/mi ni mum_i o _si ze

/sys/bl o ck/disk/q ueue/o pti mal _i o _si ze

The sg 3_uti l s package provides the sg _i nq utility, which can be used to access the BLO C K
LIMIT S VP D page. To do so, run:

164
 Chapt er 2 2 . St orage I/O Alignment and Siz e

# sg_inq -p 0xb0 disk

22.4 . St acking I/O Paramet ers

All layers of the Linux I/O stack have been engineered to propagate the various I/O parameters up the
stack. When a layer consumes an attribute or aggregates many devices, the layer must expose
appropriate I/O parameters so that upper-layer devices or tools will have an accurate view of the
storage as it transformed. Some practical examples are:

Only one layer in the I/O stack should adjust for a non-zero al i g nment_o ffset; once a layer
adjusts accordingly, it will export a device with an al i g nment_o ffset of zero.

A striped D evice Mapper (D M) device created with LVM must export a mi ni mum_i o _si ze and
o pti mal _i o _si ze relative to the stripe count (number of disks) and user-provided chunk size.

In Red Hat Enterprise Linux 7, D evice Mapper and Software Raid (MD ) device drivers can be used to
arbitrarily combine devices with different I/O parameters. The kernel's block layer will attempt to
reasonably combine the I/O parameters of the individual devices. The kernel will not prevent
combining heterogeneous devices; however, be aware of the risks associated with doing so.

For instance, a 512-byte device and a 4K device may be combined into a single logical D M device,
which would have a l o g i cal _bl o ck_si ze of 4K. File systems layered on such a hybrid device
assume that 4K will be written atomically, but in reality it will span 8 logical block addresses when
issued to the 512-byte device. Using a 4K l o g i cal _bl o ck_si ze for the higher-level D M device
increases potential for a partial write to the 512-byte device if there is a system crash.

If combining the I/O parameters of multiple devices results in a conflict, the block layer may issue a
warning that the device is susceptible to partial writes and/or is misaligned.

22.5. Logical Volume Manager

LVM provides userspace tools that are used to manage the kernel's D M devices. LVM will shift the
start of the data area (that a given D M device will use) to account for a non-zero
al i g nment_o ffset associated with any device managed by LVM. This means logical volumes will
be properly aligned (al i g nment_o ffset= 0 ).

By default, LVM will adjust for any al i g nment_o ffset, but this behavior can be disabled by setting
d ata_al i g nment_o ffset_d etecti o n to 0 in /etc/l vm/l vm. co nf. D isabling this is not
recommended.

LVM will also detect the I/O hints for a device. The start of a device's data area will be a multiple of the
mi ni mum_i o _si ze or o pti mal _i o _si ze exposed in sysfs. LVM will use the mi ni mum_i o _si ze
if o pti mal _i o _si ze is undefined (i.e. 0 ).

By default, LVM will automatically determine these I/O hints, but this behavior can be disabled by
setting d ata_al i g nment_d etecti o n to 0 in /etc/l vm/l vm. co nf. D isabling this is not
recommended.

22.6. Part it ion and File Syst em T ools

This section describes how different partition and file system management tools interact with a
device's I/O parameters.

ut il-linux-ng's libblkid and fdisk

165
St orage Administ rat ion G uide

The l i bbl ki d library provided with the uti l -l i nux-ng package includes a programmatic API to
access a device's I/O parameters. l i bbl ki d allows applications, especially those that use D irect
I/O, to properly size their I/O requests. The fd i sk utility from uti l -l i nux-ng uses l i bbl ki d to
determine the I/O parameters of a device for optimal placement of all partitions. The fd i sk utility will
align all partitions on a 1MB boundary.

part ed and libpart ed

The l i bparted library from parted also uses the I/O parameters API of l i bbl ki d . The Red Hat
Enterprise Linux 7 installer (An aco n d a) uses l i bparted , which means that all partitions created by
either the installer or parted will be properly aligned. For all partitions created on a device that does
not appear to provide I/O parameters, the default alignment will be 1MB.

The heuristics parted uses are as follows:

Always use the reported al i g nment_o ffset as the offset for the start of the first primary
partition.

If o pti mal _i o _si ze is defined (i.e. not 0 ), align all partitions on an o pti mal _i o _si ze
boundary.

If o pti mal _i o _si ze is undefined (i.e. 0 ), al i g nment_o ffset is 0 , and mi ni mum_i o _si ze
is a power of 2, use a 1MB default alignment.

This is the catch-all for " legacy" devices which don't appear to provide I/O hints. As such, by
default all partitions will be aligned on a 1MB boundary.

Note

Red Hat Enterprise Linux 7 cannot distinguish between devices that don't provide I/O hints
and those that do so with al i g nment_o ffset= 0 and o pti mal _i o _si ze= 0 . Such a
device might be a single SAS 4K device; as such, at worst 1MB of space is lost at the start
of the disk.

File Syst em t ools

The different mkfs. filesystem utilities have also been enhanced to consume a device's I/O
parameters. These utilities will not allow a file system to be formatted to use a block size smaller than
the l o g i cal _bl o ck_si ze of the underlying storage device.

Except for mkfs. g fs2, all other mkfs. filesystem utilities also use the I/O hints to layout on-disk
data structure and data areas relative to the mi ni mum_i o _si ze and o pti mal _i o _si ze of the
underlying storage device. This allows file systems to be optimally formatted for various RAID
(striped) layouts.

166
 Chapt er 2 3. Set t ing Up A Remot e Diskless Syst em

Chapter 23. Setting Up A Remote Diskless System

To set up a basic remote diskless system booted over PXE, you need the following packages:

tftp-server

xi netd

d hcp

sysl i nux

d racut-netwo rk

Note

After installing the d racut-netwo rk package, add the following line to

/etc/d racut. co nf:

add_dracutmodules+="nfs"

Remote diskless system booting requires both a tftp service (provided by tftp-server) and a
D HCP service (provided by d hcp). The tftp service is used to retrieve kernel image and i ni trd
over the network via the PXE loader.

Note

SELinux is only supported over NFSv4.2. To use SELinux, NFS must be explicitly enabled in
/etc/sysco nfi g /nfs by adding the line:

R P C NFSD AR G S= "-V 4 . 2"

Then, in /var/l i b/tftpbo o t/pxel i nux. cfg /d efaul t, change ro o t= nfs: server-
i p: /expo rted /ro o t/d i recto ry to ro o t= nfs: server-
i p: /expo rted /ro o t/d i recto ry,vers= 4 . 2.

Finally, reboot the NFS server.

The following sections outline the necessary procedures for deploying remote diskless systems in a
network environment.

Important

Some RPM packages have started using file capabilities (such as setcap and g etcap).
However, NFS does not currently support these so attempting to install or update any
packages that use file capabilities will fail.

23.1. Configuring a t ft p Service for Diskless Client s

167
St orage Administ rat ion G uide

23.1. Configuring a t ft p Service for Diskless Client s

The tftp service is disabled by default. To enable it and allow PXE booting via the network, set the
D i sabl ed option in /etc/xi netd . d /tftp to no . To configure tftp, perform the following steps:

Pro ced u re 23.1. T o co n f ig u re tftp

1. The tftp root directory (chro o t) is located in /var/l i b/tftpbo o t. Copy

/usr/share/sysl i nux/pxel i nux. 0 to /var/l i b/tftpbo o t/, as in:

cp /usr/share/sysl i nux/pxel i nux. 0 /var/l i b/tftpbo o t/

2. Create a pxel i nux. cfg directory inside the tftp root directory:

mkd i r -p /var/l i b/tftpbo o t/pxel i nux. cfg /

You will also need to configure firewall rules properly to allow tftp traffic; as tftp supports TCP
wrappers, you can configure host access to tftp via /etc/ho sts. al l o w. For more information on
configuring TCP wrappers and the /etc/ho sts. al l o w configuration file, refer to the Red Hat
Enterprise Linux 7 Security Guide; man ho sts_access also provides information about
/etc/ho sts. al l o w.

After configuring tftp for diskless clients, configure D HCP, NFS, and the exported file system
accordingly. Refer to Section 23.2, Configuring D HCP for D iskless Clients and Section 23.3,
Configuring an Exported File System for D iskless Clients for instructions on how to do so.

23.2. Configuring DHCP for Diskless Client s

After configuring a tftp server, you need to set up a D HCP service on the same host machine. Refer
to the Red Hat Enterprise Linux 7 Deployment Guide for instructions on how to set up a D HCP server.
In addition, you should enable PXE booting on the D HCP server; to do this, add the following
configuration to /etc/d hcp/d hcp. co nf:

allow booting;
allow bootp;
class "pxeclients" {
match if substring(option vendor-class-identifier, 0, 9) =
"PXEClient";
next-server server-ip;
filename "pxelinux.0";
}

Replace server-ip with the IP address of the host machine on which the tftp and D HCP services
reside. Now that tftp and D HCP are configured, all that remains is to configure NFS and the
exported file system; refer to Section 23.3, Configuring an Exported File System for D iskless Clients
for instructions.

Note

When l i bvi rt virtual machines are used as the diskless client, l i bvi rt provides the D HCP
service and the stand alone D HCP server is not used. In this situation, network booting must
be enablied with the bo o tp fi l e= ' filename' option in the l i bvi rt network
configuration, vi rsh net-ed i t.

168
 Chapt er 2 3. Set t ing Up A Remot e Diskless Syst em

23.3. Configuring an Export ed File Syst em for Diskless Client s

The root directory of the exported file system (used by diskless clients in the network) is shared via
NFS. Configure the NFS service to export the root directory by adding it to /etc/expo rts. For
instructions on how to do so, refer to Section 8.7.1, The /etc/expo rts Configuration File .

To accommodate completely diskless clients, the root directory should contain a complete Red Hat
Enterprise Linux installation. You can synchronize this with a running system via rsync, as in:

# rsync -a -e ssh --exclude='/proc/*' --exclude='/sys/*' hostname.com:/

/exported/root/directory

Replace hostname.com with the hostname of the running system with which to synchronize via
rsync. The /exported/root/directory is the path to the exported file system.

Alternatively, you can also use yum with the --i nstal l ro o t option to install Red Hat Enterprise
Linux to a specific location. For example:

yum groupinstall Base --installroot=/exported/root/directory --

releasever=/

The file system to be exported still needs to be configured further before it can be used by diskless
clients. To do this, perform the following procedure:

Pro ced u re 23.2. C o n f ig u re f ile syst em

1. Configure the exported file system's /etc/fstab to contain (at least) the following
configuration:

none /tmp tmpfs defaults 0 0

tmpfs /dev/shm tmpfs defaults 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0

2. Select the kernel that diskless clients should use (vml i nuz-kernel-version) and copy it
to the tftp boot directory:

# cp /boot/vmlinuz-kernel-version /var/lib/tftpboot/

3. Create the i ni trd (that is, i ni tramfs-kernel-version. i mg ) with network support:

# dracut initramfs-kernel-version.img kernel-version

4. The initrd's file permissions need to be changed to 600 or the pxel i nux. 0 boot loader will
fail with a " file not found" error. D o this with the following command:

# chmod go-r initramfs-kernel-version.img

5. Copy the resulting i ni tramfs-kernel-version. i mg into the tftp boot directory as well.

169
St orage Administ rat ion G uide

6. Edit the default boot configuration to use the i ni trd and kernel inside
/var/l i b/tftpbo o t. This configuration should instruct the diskless client's root to mount
the exported file system (/exported/root/directory) as read-write. To do this, configure
/var/l i b/tftpbo o t/pxel i nux. cfg /d efaul t with the following:

default rhel7

label rhel7
kernel vmlinuz-kernel-version
append initrd=initramfs-kernel-version.img root=nfs:server-
ip:/exported/root/directory rw

Replace server-ip with the IP address of the host machine on which the tftp and D HCP
services reside.

The NFS share is now ready for exporting to diskless clients. These clients can boot over the network
via PXE.

170
 Chapt er 2 4 . O nline St orage Management

Chapter 24. Online Storage Management

It is often desirable to add, remove or re-size storage devices while the operating system is running,
and without rebooting. This chapter outlines the procedures that may be used to reconfigure storage
devices on Red Hat Enterprise Linux 7 host systems while the system is running. It covers iSCSI and
Fibre Channel storage interconnects; other interconnect types may be added it the future.

This chapter focuses on adding, removing, modifying, and monitoring storage devices. It does not
discuss the Fibre Channel or iSCSI protocols in detail. For more information about these protocols,
refer to other documentation.

This chapter makes reference to various sysfs objects. Red Hat advises that the sysfs object
names and directory structure are subject to change in major Red Hat Enterprise Linux releases. This
is because the upstream Linux kernel does not provide a stable internal API. For guidelines on how
to reference sysfs objects in a transportable way, refer to the document
/usr/share/d o c/kernel -d o c-version/D o cumentati o n/sysfs-rul es. txt in the kernel
source tree for guidelines.

Warning

Online storage reconfiguration must be done carefully. System failures or interruptions during
the process can lead to unexpected results. Red Hat advises that you reduce system load to
the maximum extent possible during the change operations. This will reduce the chance of I/O
errors, out-of-memory errors, or similar errors occurring in the midst of a configuration change.
The following sections provide more specific guidelines regarding this.

In addition, Red Hat recommends that you back up all data before reconfiguring online
storage.

24 .1. T arget Set up

Red Hat Enterprise Linux 7 uses the targ etcl i shell as a front end for viewing, editing, and saving
the configuration of the Linux-IO Target without the need to manipulate the kernel target's
configuration files directly. The targ etcl i tool is a command-line interface that allows an
administrator to export local storage resources, which are backed by either files, volumes, local SCSI
devices, or RAM disks, to remote systems. The targ etcl i tool has a tree-based layout, includes
built-in tab completion, and provides full auto-complete support and inline documentation.

The hierarchy of targ etcl i does not always match the kernel interface exactly because
targ etcl i is simplified where possible.

Important

To ensure that the changes made in targ etcl i are persistent, start and enable the target
service:

# systemctl start targ et

# systemctl enabl e targ et

171
St orage Administ rat ion G uide

24 .1.1. Inst all and Run t arget cli

To install targ etcl i , use:

# yum i nstal l targ etcl i

Start the targ et service:

# systemctl start targ et

Configure targ et to start at boot time:

# systemctl enabl e targ et

Use the targ etcl i command, and then use the l s command for the layout of the tree interface:

# targ etcl i
:
/> l s
o- /........................................[...]
o- backstores.............................[...]
| o- block.................[Storage Objects: 0]
| o- fileio................[Storage Objects: 0]
| o- pscsi.................[Storage Objects: 0]
| o- ramdisk...............[Storage Ojbects: 0]
o- iscsi...........................[Targets: 0]
o- loopback........................[Targets: 0]

Note

In Red Hat Enterprise Linux 7.0, using the targ etcl i command from Bash, for example,
targ etcl i i scsi / create, does not work and does not return an error. Starting with
Red Hat Enterprise Linux 7.1, an error status code is provided to make using targ etcl i with
shell scripts more useful.

24 .1.2. Creat e a Backst ore

Backstores enable support for different methods of storing an exported LUN's data on the local
machine. Creating a storage object defines the resources the backstore uses.

Note

In Red Hat Enterprise Linux 6, the term 'backing-store' is used to refer to the mappings created.
However, to avoid confusion between the various ways 'backstores' can be used, in Red Hat
Enterprise Linux 7 the term 'storage objects' refers to the mappings created and 'backstores' is
used to describe the different types of backing devices.

The backstore devices that LIO supports are:

FILEIO ( Lin u x f ile- b acked st o rag e)

172
 Chapt er 2 4 . O nline St orage Management

FILEIO storage objects can support either wri te_back or wri te_thru operation. The
wri te_back enables the local file system cache. This improves performance but increases
the risk of data loss. It is recommended to use wri te_back= fal se to disable
wri te_back in favor of wri te_thru.

To create a fileio storage object, run the command /backsto res/fi l ei o create
file_name file_location file_size wri te_back= fal se. For example:

/> /backstores/fileio create file1 /tmp/disk1.img 200M

write_back=false
Created fileio file1 with size 209715200

B LO C K ( Lin u x B LO C K d evices)

The block driver allows the use of any block device that appears in the /sys/bl o ck to be
used with LIO. This includes physical devices (for example, HD D s, SSD s, CD s, D VD s) and
logical devices (for example, software or hardware RAID volumes, or LVM volumes).

Note

BLOCK backstores usually provide the best performance.

To create a BLOCK backstore using the /d ev/sd b block device, use the following
command:

/> /backstores/block create name=block_backend dev=/dev/sdb

Generating a wwn serial.
Created block storage object block_backend using /dev/sdb.

PSC SI ( Lin u x p ass- t h ro u g h SC SI d evices)

Any storage object that supports direct pass-through of SCSI commands without SCSI
emulation, and with an underlying SCSI device that appears with lsscsi in
/pro c/scsi /scsi (such as a SAS hard drive) can be configured as a backstore. SCSI-3
and higher is supported with this subsystem.

Warning

PSCSI should only be used by advanced users. Advanced SCSI commands such as
for Aysmmetric Logical Unit Assignment (ALUAs) or Persistent Reservations (for
example, those used by VMware ESX, and vSphere) are usually not implemented in
the device firmware and can cause malfunctions or crashes. When in doubt, use
BLOCK for production setups instead.

To create a PSCSI backstore for a physical SCSI device, a T Y P E_R O M device using
/d ev/sr0 in this example, use:

/> backstores/pscsi/ create name=pscsi_backend dev=/dev/sr0

Generating a wwn serial.
Created pscsi storage object pscsi_backend using /dev/sr0

173
St orage Administ rat ion G uide

Memo ry C o p y R AM d isk ( Lin u x R AMD ISK _MC P)

Memory Copy RAM disks (ramd i sk) provide RAM disks with full SCSI emulation and
separate memory mappings using memory copy for initiators. This provides capability for
multi-sessions and is particularly useful for fast, volatile mass storage for production
purposes.

To create a 1GB RAM disk backstore, use the following command:

/> backstores/ramdisk/ create name=rd_backend size=1GB

Generating a wwn serial.
Created rd_mcp ramdisk rd_backend with size 1GB.

24 .1.3. Creat e an iSCSI T arget

To create an iSCSI target:

Pro ced u re 24 .1. C reat e an iSC SI t arg et

1. Run targ etcl i .

2. Move into the iSCSI configuration path:

/> iscsi/

Note

The cd command is also accepted to change directories, as well as simply listing the
path to move into.

3. Create an iSCSI target using a default target name.

/iscsi> create
Created target
iqn.2003-01.org.linux-iscsi.hostname.x8664:sn.78b473f296ff
Created TPG1

Or create an iSCSI target using a specified name.

/iscsi > create iqn.2006-04.com.example:444

Created target iqn.2006-04.com.example:444
Created TPG1

4. Verify that the newly created target is visible when targets are listed with l s.

/iscsi > ls
o- iscsi.......................................[1 Target]
o- iqn.2006-04.com.example:444................[1 TPG]
o- tpg1...........................[enabled, auth]
o- acls...............................[0 ACL]
o- luns...............................[0 LUN]
o- portals.........................[0 Portal]

174
 Chapt er 2 4 . O nline St orage Management

Note

As of Red Hat Enterprise Linux 7.1, whenever a target is created, a default portal is also
created.

24 .1.4 . Configure an iSCSI Port al

To configure an iSCSI portal, an iSCSI target must first be created and associated with a TPG. For
instructions on how to do this, refer to Section 24.1.3, Create an iSCSI Target .

Note

As of Red Hat Enterprise Linux 7.1 when an iSCSI target is created, a default portal is created
as well. This portal is set to listen on all IP addresses with the default port number (that is,
0.0.0.0:3260). To remove this and add only specified portals, use /i scsi /iqn-
name/tpg 1/po rtal s d el ete i p_ad d ress= 0 . 0 . 0 . 0 i p_po rt= 326 0 then create a
new portal with the required information.

Pro ced u re 24 .2. C reat e an iSC SI Po rt al

1. Move into the TPG.

/iscsi> iqn.2006-04.example:444/tpg1/

2. There are two ways to create a portal: create a default portal, or create a portal specifying
what IP address to listen to.

Creating a default portal uses the default iSCSI port 3260 and allows the target to listen on
all IP addresses on that port.

/iscsi/iqn.20...mple:444/tpg1> portals/ create

Using default IP port 3260
Binding to INADDR_Any (0.0.0.0)
Created network portal 0.0.0.0:3260

To create a portal specifying what IP address to listen to, use the following command.

/iscsi/iqn.20...mple:444/tpg1> portals/ create 192.168.122.137

Using default IP port 3260
Created network portal 192.168.122.137:3260

3. Verify that the newly created portal is visible with the l s command.

/iscsi/iqn.20...mple:444/tpg1> ls
o- tpg.................................. [enambled, auth]
o- acls ......................................[0 ACL]
o- luns ......................................[0 LUN]
o- portals ................................[1 Portal]
o- 192.168.122.137:3260......................[OK]

175
St orage Administ rat ion G uide

24 .1.5. Configure LUNs

To configure LUNs, first create storage objects. See Section 24.1.2, Create a Backstore for more
information.

Pro ced u re 24 .3. C o n f ig u re LU N s

1. Create LUNs of already created storage objects.

/iscsi/iqn.20...mple:444/tpg1> luns/ create

/backstores/ramdisk/ramdisk1
Created LUN 0.

/iscsi/iqn.20...mple:444/tpg1> luns/ create /backstores/block/block1

Created LUN 1.

/iscsi/iqn.20...mple:444/tpg1> luns/ create /backstores/fileio/file1

Created LUN 2.

2. Show the changes.

/iscsi/iqn.20...mple:444/tpg1> ls
o- tpg.................................. [enambled, auth]
o- acls ......................................[0 ACL]
o- luns .....................................[3 LUNs]
| o- lun0.........................[ramdisk/ramdisk1]
| o- lun1.................[block/block1 (/dev/vdb1)]
| o- lun2...................[fileio/file1 (/foo.img)]
o- portals ................................[1 Portal]
o- 192.168.122.137:3260......................[OK]

Note

Be aware that the default LUN name starts at 0, as opposed to 1 as was the case when
using tg td in Red Hat Enterprise Linux 6.

Important

By default, LUNs are created with read-write permissions. In the event that a new LUN is added
after ACLs have been created that LUN will be automatically mapped to all available ACLs.
This can cause a security risk. Use the following procedure to create a LUN as read-only.

Pro ced u re 24 .4 . C reat e a read - o n ly LU N

1. To create a LUN with read-only permissions, first use the following command:

/> set global auto_add_mapped_luns=false

Parameter auto_add_mapped_luns is now 'false'.

176
 Chapt er 2 4 . O nline St orage Management

This prevents the auto mapping of LUNs to existing ACLs allowing the manual mapping of
LUNs.

2. Next, manually create the LUN with the command

i scsi /target_iqn_name/tpg 1/acl s/initiator_iqn_name/ create
mapped _l un= next_sequential_LUN_number tpg _l un_o r_backsto re= backstore
wri te_pro tect= 1.

/> iscsi/iqn.2015-06.com.redhat:target/tpg1/acls/iqn.2015-
06.com.redhat:initiator/ create mapped_lun=1
tpg_lun_or_backstore=/backstores/block/block2 write_protect=1
Created LUN 1.
Created Mapped LUN 1.
/> ls
o- / ...................................................... [...]
o- backstores ........................................... [...]
<snip>
o- iscsi ......................................... [Targets: 1]
| o- iqn.2015-06.com.redhat:target .................. [TPGs: 1]
| o- tpg1 ............................ [no-gen-acls, no-auth]
| o- acls ....................................... [ACLs: 2]
| | o- iqn.2015-06.com.redhat:initiator .. [Mapped LUNs: 2]
| | | o- mapped_lun0 .............. [lun0 block/disk1 (rw)]
| | | o- mapped_lun1 .............. [lun1 block/disk2 (ro)]
| o- luns ....................................... [LUNs: 2]
| | o- lun0 ...................... [block/disk1 (/dev/vdb)]
| | o- lun1 ...................... [block/disk2 (/dev/vdc)]
<snip>

The mapped_lun1 line now has (ro) at the end (unlike mapped_lun0's (rw)) stating that it is
read-only.

24 .1.6. Configure ACLs

Create an ACL for each initiator that will be connecting. This enforces authentication when that
initiator connects, allowing only LUNs to be exposed to each initiator. Usually each initator has
exclusive access to a LUN. Both targets and initiators have unique identifying names. The initiator's
unique name must be known to configure ACLs. For open-iscsi initiators, this can be found in
/etc/i scsi /i ni ti ato rname. i scsi .

Pro ced u re 24 .5. C o n f ig u re AC Ls

1. Move into the acls directory.

/iscsi/iqn.20...mple:444/tpg1> acls/

2. Create an ACL. Either use the initiator name found in

/etc/i scsi /i ni ti ato rname. i scsi on the initiator, or if using a name that is easier to
remember, refer to Section 24.2, Creating an iSCSI Initiator to ensure ACL matches the
initiator. For example:

/iscsi/iqn.20...444/tpg1/acls> create iqn.2006-

04.com.example.foo:888
Created Node ACL for iqn.2006-04.com.example.foo:888
Created mapped LUN 2.

177
St orage Administ rat ion G uide

Created mapped LUN 1.

Created mapped LUN 0.

Note

The above example's behavior depends on the setting used. In this case, the global
setting auto _ad d _mapped _l uns is used. This automatically maps LUNs to any
created ACL.

3. Show the changes.

/iscsi/iqn.20...444/tpg1/acls> ls
o- acls .................................................[1 ACL]
o- iqn.2006-04.com.example.foo:888 ....[3 Mapped LUNs, auth]
o- mapped_lun0 .............[lun0 ramdisk/ramdisk1 (rw)]
o- mapped_lun1 .................[lun1 block/block1 (rw)]
o- mapped_lun2 .................[lun2 fileio/file1 (rw)]

24 .1.7. Fibre Channel over Et hernet (FCoE) T arget Set up

In addition to mounting LUNs over FCoE, as described in Section 24.4, Configuring a Fibre Channel
over Ethernet Interface , exporting LUNs to other machines over FCoE is also supported with the aid
of targ etcl i .

Important

Before proceeding, refer to Section 24.4, Configuring a Fibre Channel over Ethernet Interface
and verify that basic FCoE setup is completed, and that fco ead m -i displays configured
FCoE interfaces.

Pro ced u re 24 .6 . C o n f ig u re FC o E t arg et

1. Setting up an FCoE target requires the installation of the targ etcl i package, along with its
dependencies. Refer to Section 24.1, Target Setup for more information on targ etcl i
basics and set up.

2. Create an FCoE target instance on an FCoE interface.

/> tcm_fc/ create 00:11:22:33:44:55:66:77

If FCoE interfaces are present on the system, tab-completing after create will list available
interfaces. If not, ensure fco ead m -i shows active interfaces.

3. Map a backstore to the target instance.

Examp le 24 .1. Examp le o f map p in g a b ackst o re t o t h e t arg et in st an ce

/> tcm_fc/00:11:22:33:44:55:66:77

178
 Chapt er 2 4 . O nline St orage Management

/> l uns/ create /backsto res/fi l ei o /example2

4. Allow access to the LUN from an FCoE initiator.

/> acl s/ create 00:99:88:77:66:55:44:33

The LUN should now be accessible to that initiator.

5. To make the changes persistant across reboots, use the saveco nfi g command and type
yes when prompted. If this is not done the configuration will be lost after rebooting.

6. Exit targ etcl i by typing exi t or entering ctrl +D .

24 .1.8. Remove object s wit h targ etcl i

To remove an backstore use the command:

/> /backstores/backstore-type/backstore-name

To remove parts of an iSCSI target, such as an ACL, use the following command:

/> /iscsi/iqn-name/tpg/acls/ delete iqn-name

To remove the entire target, including all ACLs, LUNs, and portals, use the following command:

/> /iscsi delete iqn-name

24 .1.9. targ etcl i References

For more information on targ etcl i , refer to the following resources:

man targ etcl i

The targ etcl i man page. It includes an example walk through.

T h e Lin u x SC SI T arg et Wiki

http://linux-iscsi.org/wiki/Targetcli

Screen cast b y An d y G ro ver

https://www.youtube.com/watch?v=BkBGTBadOO8

Note

This was uploaded on February 28, 2012. As such, the service name has changed
from targ etcl i to targ et.

24 .2. Creat ing an iSCSI Init iat or

After creating a target with targ etcl i as in Section 24.1, Target Setup , use the i scsi ad m utility

179
St orage Administ rat ion G uide

to set up an initiator.

In Red Hat Enterprise Linux 7, the iSCSI service is lazily started by default: the service starts after
running the i scsi ad m command.

Pro ced u re 24 .7. C reat in g an iSC SI In it iat o r

1. Install i scsi -i ni ti ato r-uti l s:

# yum i nstal l i scsi -i ni ti ato r-uti l s

2. If the ACL was given a custom name in Section 24.1.6, Configure ACLs , modify the
/etc/i scsi /i ni ti ato rname. i sci file accordingly. For example:

# cat /etc/i scsi /i ni ti ato rname. i scsi

InitiatorName=iqn.2006-04.com.example.node1

3. D iscover the target:

# i scsi ad m -m d i sco very -t st -p target-ip-address

10.64.24.179:3260,1 iqn.2006-04.com.example:3260

4. Set user-created ACLs within the TPG node on the target server:

/i scsi /i q n. 20 . . . scsi : 4 4 4 /tpg 1> set attri bute

g enerate_no d e_acl s= 1

Then, use the exi t command.

5. Log in to the target with the target IQN you discovered in step 3:

# i scsi ad m -m no d e -T iqn.2006-04.com.example:3260 -l
Logging in to [iface: default, target: iqn.2006-
04.com.example:3260, portal: 10.64.24.179,3260] (multiple)
Login to [iface: default, target: iqn.2006-04.com.example:3260,
portal: 10.64.24.179,3260] successful.

This procedure can be followed for any number of initators connected to the same LUN so long as
their specific initiator names are added to the ACL as described in Section 24.1.6, Configure ACLs .

24 .3. Fibre Channel

This section discusses the Fibre Channel API, native Red Hat Enterprise Linux 7 Fibre Channel
drivers, and the Fibre Channel capabilities of these drivers.

24 .3.1. Fibre Channel API

Below is a list of /sys/cl ass/ directories that contain files used to provide the userspace API. In
each item, host numbers are designated by H, bus numbers are B, targets are T, logical unit numbers
(LUNs) are L, and remote port numbers are R.

180
 Chapt er 2 4 . O nline St orage Management

Important

If your system is using multipath software, Red Hat recommends that you consult your
hardware vendor before changing any of the values described in this section.

T ran sp o rt : /sys/cl ass/fc_transpo rt/targ etH: B: T/

po rt_i d 24-bit port ID /address

no d e_name 64-bit node name

po rt_name 64-bit port name

R emo t e Po rt : /sys/cl ass/fc_remo te_po rts/rpo rt-H: B-R/

po rt_i d

no d e_name

po rt_name

d ev_l o ss_tmo number of seconds to wait before marking a link as " bad" . Once a
link is marked bad, I/O running on its corresponding path (along with any new I/O on
that path) will be failed.

The default d ev_l o ss_tmo value varies, depending on which driver/device is used. If a
Qlogic adapter is used, the default is 35 seconds, while if an Emulex adapter is used, it
is 30 seconds. The d ev_l o ss_tmo value can be changed via the
scsi _transpo rt_fc module parameter d ev_l o ss_tmo , although the driver can
override this timeout value.

The maximum d ev_l o ss_tmo value is 600 seconds. If d ev_l o ss_tmo is set to zero or
any value greater than 600, the driver's internal timeouts will be used instead.

fast_i o _fai l _tmo length of time to wait before failing I/O executed when a link
problem is detected. I/O that reaches the driver will fail. If I/O is in a blocked queue, it will
not be failed until d ev_l o ss_tmo expires and the queue is unblocked.

H o st : /sys/cl ass/fc_ho st/ho stH/

po rt_i d

i ssue_l i p instructs the driver to rediscover remote ports.

24 .3.2. Nat ive Fibre Channel Drivers and Capabilit ies

Red Hat Enterprise Linux 7 ships with the following native Fibre Channel drivers:

l pfc

q l a2xxx

zfcp

bfa

181
St orage Administ rat ion G uide

Important

The q la2xxx driver runs in initiator mode by default. To use qla2xxx with Linux-IO, enable
Fibre Channel target mode with the corresponding qlini_mode module parameter.

First, make sure that the firmware package for your qla device, such as ql2200-firmware or
similar, is installed.

To enable target mode, add the following parameter to the

/usr/l i b/mo d pro be. d /q l a2xxx. co nf qla2xxx module configuration file:

options qla2xxx qlini_mode=disabled

Then, use the d racut -f command to rebuild the initial ramdisk (i ni trd ), and reboot the
system for the changes to take effect.

Table 24.1, Fibre Channel API Capabilities describes the different Fibre Channel API capabilities of
each native Red Hat Enterprise Linux 7 driver. X denotes support for the capability.

T ab le 24 .1. Fib re C h an n el API C ap ab ilit ies

l pfc q l a2xxx zfcp bfa

Transport X X X X
po rt_i d
Transport X X X X
no d e_name
Transport X X X X
po rt_name
Remote Port X X X X
d ev_l o ss_tmo
Remote Port X X [a] X [b ] X
fast_i o _fai l _
tmo
Host po rt_i d X X X X
Host i ssue_l i p X X X
[a] Sup p o rted as o f Red Hat Enterp ris e Linux 5.4
[b ] Sup p o rted as o f Red Hat Enterp ris e Linux 6 .0

24 .4 . Configuring a Fibre Channel over Et hernet Int erface

Setting up and deploying a Fibre Channel over Ethernet (FCoE) interface requires two packages:

fco e-uti l s

l l d pad

Once these packages are installed, perform the following procedure to enable FCoE over a virtual
LAN (VLAN):

Pro ced u re 24 .8. C o n f ig u rin g an Et h ern et in t erf ace t o u se FC o E

182
 Chapt er 2 4 . O nline St orage Management

1. To configure a new VLAN, make a copy of an existing network script, for example
/etc/fco e/cfg -eth0 , and change the name to the Ethernet device that supports FCoE.
This provides you with a default file to configure. Given that the FCoE device is ethX, run:

# cp /etc/fcoe/cfg-ethx /etc/fcoe/cfg-ethX

Modify the contents of cfg -ethX as needed. Notably, set D C B_R EQ UIR ED to no for
networking interfaces that implement a hardware D ata Center Bridging Exchange (D CBX)
protocol client.

2. If you want the device to automatically load during boot time, set O NBO O T = yes in the
corresponding /etc/sysco nfi g /netwo rk-scri pts/i fcfg -ethX file. For example, if the
FCoE device is eth2, edit /etc/sysco nfi g /netwo rk-scri pts/i fcfg -eth2 accordingly.

3. Start the data center bridging daemon (d cbd ) by running:

# systemctl start lldpad

4. For networking interfaces that implement a hardware D CBX client, skip this step.

For interfaces that require a software D CBX client, enable data center bridging on the
Ethernet interface by running:

# dcbtool sc ethX dcb on

Then, enable FCoE on the Ethernet interface by running:

# dcbtool sc ethX app:fcoe e:1

Note that these commands only work if the d cbd settings for the Ethernet interface were not
changed.

5. Load the FCoE device now using:

# ip link set dev ethX up

6. Start FCoE using:

# systemctl start fcoe

The FCoE device appears soon if all other settings on the fabric are correct. To view
configured FCoE devices, run:

# fcoeadm -i

After correctly configuring the Ethernet interface to use FCoE, Red Hat recommends that you set FCoE
and l l d pad to run at startup. To do so, use chkco nfi g , as in:

# systemctl enable lldpad

# systemctl enable fcoe

183
St orage Administ rat ion G uide

Note

Running the # systemctl sto p fco e command stops the daemon, but does not reset the
configuration of FCoE interfaces. To do so, run the # systemctl -s SIG HUP ki l l fco e
command.

As of Red Hat Enterprise Linux 7, Network Manager has the ability to query and set the D CB settings
of a D CB capable Ethernet interface.

24 .5. Configuring an FCoE Int erface t o Aut omat ically Mount at Boot

Note

The instructions in this section are available in /usr/share/d o c/fco e-

uti l s-version/R EAD ME as of Red Hat Enterprise Linux 6.1. Refer to that document for any
possible changes throughout minor releases.

You can mount newly discovered disks via ud ev rules, auto fs, and other similar methods.
Sometimes, however, a specific service might require the FCoE disk to be mounted at boot-time. In
such cases, the FCoE disk should be mounted as soon as the fco e service runs and before the
initiation of any service that requires the FCoE disk.

To configure an FCoE disk to automatically mount at boot, add proper FCoE mounting code to the
startup script for the fco e service. The fco e startup script is /etc/i ni t. d /fco e.

The FCoE mounting code is different per system configuration, whether you are using a simple
formatted FCoE disk, LVM, or multipathed device node.

Examp le 24 .2. FC o E mo u n t in g co d e

The following is a sample FCoE mounting code for mounting file systems specified via wild cards
in /etc/fstab:

mount_fcoe_disks_from_fstab()
{
local timeout=20
local done=1
local fcoe_disks=($(egrep 'by-path\/fc-.*_netdev' /etc/fstab | cut
-d ' ' -f1))

test -z $fcoe_disks & & return 0

echo -n "Waiting for fcoe disks . "

while [ $timeout -gt 0 ]; do
for disk in ${fcoe_disks[*]}; do
if ! test -b $disk; then
done=0
break
fi
done

184
 Chapt er 2 4 . O nline St orage Management

test $done -eq 1 & & break;

sleep 1
echo -n ". "
done=1
let timeout--
done

if test $timeout -eq 0; then

echo "timeout!"
else
echo "done!"
fi

# mount any newly discovered disk

mount -a 2>/dev/null
}

The mo unt_fco e_d i sks_fro m_fstab function should be invoked after the fco e service script
starts the fco emo n daemon. This will mount FCoE disks specified by the following paths in
/etc/fstab:

/dev/disk/by-path/fc-0xXX:0xXX /mnt/fcoe-disk1 ext3 defaults,_netdev

0 0
/dev/disk/by-path/fc-0xYY:0xYY /mnt/fcoe-disk2 ext3 defaults,_netdev
0 0

Entries with fc- and _netd ev sub-strings enable the mo unt_fco e_d i sks_fro m_fstab function
to identify FCoE disk mount entries. For more information on /etc/fstab entries, refer to man 5
fstab.

Note

The fco e service does not implement a timeout for FCoE disk discovery. As such, the FCoE
mounting code should implement its own timeout period.

24 .6. iSCSI

This section describes the iSCSI API and the i scsi ad m utility. Before using the i scsi ad m utility,
install the i scsi -i ni ti ato r-uti l s package first by running yum i nstal l i scsi -
i ni ti ato r-uti l s.

In Red Hat Enterprise Linux 7, the iSCSI service is lazily started by default. If root is not on an iSCSI
device or there are no nodes marked with no d e. startup = auto mati c then the iSCSI service will
not start until an i scsi ad m command is run that requires iscsid or the iscsi kernel modules to be
started. For example, running the discovery command i scsi ad m -m d i sco very -t st -p
i p: po rt will cause iscsiadmin to start the iSCSI service.

To force the iscsid daemon to run and iSCSI kernel modules to load, run servi ce i scsi d
fo rce-start.

24 .6.1. iSCSI API

185
St orage Administ rat ion G uide

To get information about running sessions, run:

# iscsiadm -m session -P 3

This command displays the session/device state, session ID (sid), some negotiated parameters, and
the SCSI devices accessible through the session.

For shorter output (for example, to display only the sid-to-node mapping), run:

# iscsiadm -m session -P 0

or

# iscsiadm -m session

These commands print the list of running sessions with the format:

driver [sid] target_ip:port,target_portal_group_tag proper_target_name

Examp le 24 .3. O u t p u t o f t h e i sci sad m -m sessi o n co mman d

For example:

# iscsiadm -m session

tcp [2] 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311

tcp [3] 10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311

For more information about the iSCSI API, refer to /usr/share/d o c/i scsi -i ni ti ato r-
uti l s-version/R EAD ME.

24 .7. Persist ent Naming

Red Hat Enterprise Linux provides a number of ways to identify storage devices. It is important to use
the correct option to identify each device when used in order to avoid inadvertently accessing the
wrong device, particularly when installing to or reformatting drives.

24 .7.1. /d ev/sd * and t heir Major and Minor Numbers

Storage devices managed by the sd driver are identified internally by a collection of major device
numbers and their associated minor numbers. The major device numbers used for this purpose are
not in a contiguous range. Each storage device is represented by a major number and a range of
minor numbers, which are used to identify either the entire device or a partition within the device.
There is a direct association between the major and minor numbers allocated to a device and
numbers in the form of sd <letter(s)><optional number(s)>. Whenever the sd driver detects a
new device, an available major number and minor number range is allocated. Whenever a device is
removed from the operating system, the major number and minor number range is freed for later
reuse.

The major and minor number range and associated sd names are allocated for each device when it
is detected. This means that the association between the major and minor number range and

186
 Chapt er 2 4 . O nline St orage Management

associated sd names can change if the order of device detection changes. Although this is unusual
with some hardware configurations (for example, with an internal SCSI controller and disks that have
thier SCSI target ID assigned by their physical location within a chassis), it can nevertheless occur.
Examples of situations where this can happen are as follows:

A disk may fail to power up or respond to the SCSI controller. This will result in it not being
detected by the normal device proble. The disk will not be accessible to the system and
subsequent devices will have their major and minor number range, including the associated sd
names shifted down. For example, should a disk normally referred to as sd b is not detected, a
disk that is normally referred to as sd c would instead appear as sd b.

A SCSI controller (host bus adapter, or HBA) may fail to initialize, causing all disks connected to
that HBA to not be detected. Any disks connected to subsequently probed HBAs would be
assigned different major and minor number ranges, and different associated sd names.

The order of driver initialization could change if different types of HBAs are present in the system.
This would cause the disks connected to those HBAs to be detected in a different order. This can
also occur if HBAs are moved to different PCI slots on the system.

D isks connected to the system with Fibre Channel, iSCSI, or FCoE adapters might be inaccessible
at the time the storage devices are probed, due to a storage array or intervening switch being
powered off, for example. This could occur when a system reboots after a power failure, if the
storage array takes longer to come online than the system take to boot. Although some Fibre
Channel drivers support a mechanism to specify a persistent SCSI target ID to WWPN mapping,
this will not cause the major and minor number ranges, and the associated sd names to be
reserved, it will only provide consistent SCSI target ID numbers.

These reasons make it undesireable to use the major and minor number range or the associated sd
names when referring to devices, such as in the /etc/fstab file. There is the possibility that the
wrong device will be mounted and data corruption could result.

Occassionally, however, it is still necessary to refer to the sd names even when another mechanism
is used (such as when errors are reported by a device). This is because the Linux kernel uses sd
names (and also SCSI host/channel/target/LUN tuples) in kernel messages regarding the device.

24 .7.2. WWID

The World Wide Identifier (WWID ) can be used in reliably identifying devices. It is a persistent, system-
independent ID that the SCSI Standard requires from all SCSI devices. The WWID identifier is
guaranteed to be unique for every storage device, and independent of the path that is used to access
the device.

This identifier can be obtained by issuing a SCSI Inquiry to retrieve the Device Identification Vital
Product Data (page 0 x83) or Unit Serial Number (page 0 x80 ). The mappings from these WWID s to the
current /d ev/sd names can be seen in the symlinks maintained in the /d ev/d i sk/by-i d /
directory.

Examp le 24 .4 . WWID

For example, a device with a page 0 x83 identifier would have:

scsi-3600508b400105e210000900000490000 -> ../../sda

Or, a device with a page 0 x80 identifier would have:

scsi-SSEAGATE_ST373453LW_3HW1RHM6 -> ../../sda

187
St orage Administ rat ion G uide

Red Hat Enterprise Linux automatically maintains the proper mapping from the WWID -based device
name to a current /d ev/sd name on that system. Applications can use the /d ev/d i sk/by-i d /
name to reference the data on the disk, even if the path to the device changes, and even when
accessing the device from different systems.

If there are multiple paths from a system to a device, d evice- map p er- mu lt ip at h uses the WWID to
detect this. D evice- map p er- mu lt ip at h then presents a single " pseudo-device" in
/d ev/mapper/wwi d , such as /d ev/mapper/36 0 0 50 8b4 0 0 10 5d f70 0 0 0 e0 0 0 0 0 ac0 0 0 0 .

The command mul ti path -l shows the mapping to the non-persistent identifiers:
Host: Channel: Target: LUN, /d ev/sd name, and the majo r: mi no r number.

3600508b400105df70000e00000ac0000 dm-2 vendor,product

[size=20G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 5:0:1:1 sdc 8:32 [active][undef]
\_ 6:0:1:1 sdg 8:96 [active][undef]
\_ round-robin 0 [prio=0][enabled]
\_ 5:0:0:1 sdb 8:16 [active][undef]
\_ 6:0:0:1 sdf 8:80 [active][undef]

D evice- map p er- mu lt ip at h automatically maintains the proper mapping of each WWID -based
device name to its corresponding /d ev/sd name on the system. These names are persistent across
path changes, and they are consistent when accessing the device from different systems.

When the user_fri end l y_names feature (of d evice- map p er- mu lt ip at h ) is used, the WWID is
mapped to a name of the form /d ev/mapper/mpathn. By default, this mapping is maintained in the
file /etc/mul ti path/bi nd i ng s. These mpathn names are persistent as long as that file is
maintained.

Important

If you use user_fri end l y_names, then additional steps are required to obtain consistent
names in a cluster. Refer to the Consistent Multipath D evice Names in a Cluster section in the
Using DM Multipath Configuration and Administration book.

In addition to these persistent names provided by the system, you can also use ud ev rules to
implement persistent names of your own, mapped to the WWID of the storage.

24 .7.3. Device Names Managed by t he ud ev mechanism ( /d ev/d i sk/by-* )

The ud ev mechanism consists of three major components:

T h e kern el

Generates events that are sent to user space when devices are added, removed, or
changed.

T h e ud evd d aemo n

Receives the events.

T h e ud ev ru les

188
 Chapt er 2 4 . O nline St orage Management

Specifies the action to take when the ud ev daemon receives the kernel events.

This mechanism is used for all types of devices in Linux, not just for storage devices. In the case of
storage devices, Red Hat Enterprise Linux contains ud ev rules that create symbolic links in the
/d ev/d i sk/ directory allowing storage devices to be referred to by their contents, a unique
identifier, their serial number, or the hardware path used to access the device.

/d ev/d i sk/by-l abel

Entries in this directory provide a symbolic name that refers to the storage device by a label
in the contents (that is, the data) stored on the device. The bl ki d program is used to read
data from the device and determine a name (that is, a label) for the device. For example:

/dev/disk/by-label/Boot

Note

The information is obtained from the contents (that is, the data) on the device so if
the contents are copied to another device, the label will remain the same.

The label can also be used to refer to the device in /etc/fstab using the following syntax:

LABEL=Boot

/d ev/d i sk/by-uui d

Entries in this directory provide a symbolic name that refers to the storage device by a
unique identifier in the contents (that is, the data) stored on the device. The bl ki d program
is used to read data from the device and obtain a unique identifier (that is, the uuid) for the
device. For example:

UUID=3e6be9de-8139-11d1-9106-a43f08d823a6

/d ev/d i sk/by-i d

Entries in this directory provide a symbolic name that refers to the storage device by a
unique identifier (different from all other storage devices). The identifier is a property of the
device but is not stored in the contents (that is, the data) on the devices. For example:

/dev/disk/by-id/scsi-3600508e000000000ce506dc50ab0ad05

/dev/disk/by-id/wwn-0x600508e000000000ce506dc50ab0ad05

The id is obtained from the world-wide ID of the device, or the device serial number. The
/d ev/d i sk/by-i d entries may also include a partition number. For example:

/dev/disk/by-id/scsi-3600508e000000000ce506dc50ab0ad05-part1

/dev/disk/by-id/wwn-0x600508e000000000ce506dc50ab0ad05-part1

/d ev/d i sk/by-path

189
St orage Administ rat ion G uide

Entries in this directory provide a symbolic name that refers to the storage device by the
hardware path used to access the device, beginning with a reference to the storage
controller in the PCI hierachy, and including the SCSI host, channel, target, and LUN
numbers and, optionally, the partition number. Although these names are preferable to
using major and minor numbers or sd names, caution must be used to ensure that the
target numbers do not change in a Fibre Channel SAN environment (for example, through
the use of persistent binding) and that the use of the names is updated if a host adapter is
moved to a different PCI slot. In addition, there is the possibility that the SCSI host numbers
could change if a HBA fails to probe, if drivers are loaded in a different order, or if a new
HBA is installed on the system. An example of by-path listing is:

/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0

The /d ev/d i sk/by-path entries may also include a partition number, such as:

/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0-part1

2 4 .7 .3.1 . Lim it at io ns o f t he ud ev De vice Nam ing Co nve nt io n

The following are some limitations of the ud ev naming convention.

It is possible that the device may not be accessible at the time the query is performed because the
ud ev mechanism may rely on the ability to query the storage device when the ud ev rules are
processed for a ud ev event. This is more likely to occur with Fibre Channel, iSCSI or FCoE
storage devices when the device is not located in the server chassis.

The kernel may also send ud ev events at any time, causing the rules to be processed and
possibly causing the /d ev/d i sk/by-* links to be removed if the device is not accessible.

There can be a delay between when the ud ev event is generated and when it is processed (such
as when a large number of devices are detected and the user-space ud evd daemon takes some
amount of time to process the rules for each one). This could cause a delay between when the
kernel detects the device and when the /d ev/d i sk/by-* names are available.

External programs such as bl ki d invoked by the rules may open the device for a brief period of
time, making the device inaccessible for other uses.

24 .8. Removing a St orage Device

Before removing access to the storage device itself, it is advisable to back up data from the device
first. Afterwards, flush I/O and remove all operating system references to the device (as described
below). If the device uses multipathing, then do this for the multipath " pseudo device" (Section 24.7.2,
WWID ) and each of the identifiers that represent a path to the device. If you are only removing a
path to a multipath device, and other paths will remain, then the procedure is simpler, as described in
Section 24.10, Adding a Storage D evice or Path .

Removal of a storage device is not recommended when the system is under memory pressure, since
the I/O flush will add to the load. To determine the level of memory pressure, run the command
vmstat 1 10 0 ; device removal is not recommended if:

Free memory is less than 5% of the total memory in more than 10 samples per 100 (the command
free can also be used to display the total memory).

Swapping is active (non-zero si and so columns in the vmstat output).

The general procedure for removing all access to a device is as follows:

190
 Chapt er 2 4 . O nline St orage Management

Pro ced u re 24 .9 . En su rin g a C lean D evice R emo val

1. Close all users of the device and backup device data as needed.

2. Use umo unt to unmount any file systems that mounted the device.

3. Remove the device from any md and LVM volume using it. If the device is a member of an LVM
Volume group, then it may be necessary to move data off the device using the pvmo ve
command, then use the vg red uce command to remove the physical volume, and (optionally)
pvremo ve to remove the LVM metadata from the disk.

4. If the device uses multipathing, run mul ti path -l and note all the paths to the device.
Afterwards, remove the multipathed device using mul ti path -f d evi ce.

5. Run bl o ckd ev --fl ushbufs device to flush any outstanding I/O to all paths to the
device. This is particularly important for raw devices, where there is no umo unt or vg red uce
operation to cause an I/O flush.

6. Remove any reference to the device's path-based name, like /d ev/sd , /d ev/d i sk/by-
path or the majo r: mi no r number, in applications, scripts, or utilities on the system. This is
important in ensuring that different devices added in the future will not be mistaken for the
current device.

7. Finally, remove each path to the device from the SCSI subsystem. To do so, use the command
echo 1 > /sys/bl o ck/device-name/d evi ce/d el ete where device-name may be
sd e, for example.

Another variation of this operation is echo 1 >

/sys/cl ass/scsi _d evi ce/h: c: t: l/d evi ce/d el ete, where h is the HBA number, c is
the channel on the HBA, t is the SCSI target ID , and l is the LUN.

Note

The older form of these commands, echo "scsi remo ve-si ng l e-d evi ce 0 0 0
0 " > /pro c/scsi /scsi , is deprecated.

You can determine the device-name, HBA number, HBA channel, SCSI target ID and LUN for a
device from various commands, such as l sscsi , scsi _i d , mul ti path -l , and l s -l
/d ev/d i sk/by-*.

After performing Procedure 24.9, Ensuring a Clean D evice Removal , a device can be physically
removed safely from a running system. It is not necessary to stop I/O to other devices while doing so.

Other procedures, such as the physical removal of the device, followed by a rescan of the SCSI bus
(as described in Section 24.11, Scanning Storage Interconnects ) to cause the operating system
state to be updated to reflect the change, are not recommended. This will cause delays due to I/O
timeouts, and devices may be removed unexpectedly. If it is necessary to perform a rescan of an
interconnect, it must be done while I/O is paused, as described in Section 24.11, Scanning Storage
Interconnects .

24 .9. Removing a Pat h t o a St orage Device

If you are removing a path to a device that uses multipathing (without affecting other paths to the
device), then the general procedure is as follows:

191
St orage Administ rat ion G uide

Pro ced u re 24 .10. R emo vin g a Pat h t o a St o rag e D evice

1. Remove any reference to the device's path-based name, like /d ev/sd or /d ev/d i sk/by-
path or the majo r: mi no r number, in applications, scripts, or utilities on the system. This is
important in ensuring that different devices added in the future will not be mistaken for the
current device.

2. Take the path offline using echo o ffl i ne > /sys/bl o ck/sd a/d evi ce/state.

This will cause any subsequent I/O sent to the device on this path to be failed immediately.
D evice- map p er- mu lt ip at h will continue to use the remaining paths to the device.

3. Remove the path from the SCSI subsystem. To do so, use the command echo 1 >
/sys/bl o ck/device-name/d evi ce/d el ete where device-name may be sd e, for
example (as described in Procedure 24.9, Ensuring a Clean D evice Removal ).

After performing Procedure 24.10, Removing a Path to a Storage D evice , the path can be safely
removed from the running system. It is not necessary to stop I/O while this is done, as d evice-
map p er- mu lt ip at h will re-route I/O to remaining paths according to the configured path grouping
and failover policies.

Other procedures, such as the physical removal of the cable, followed by a rescan of the SCSI bus to
cause the operating system state to be updated to reflect the change, are not recommended. This will
cause delays due to I/O timeouts, and devices may be removed unexpectedly. If it is necessary to
perform a rescan of an interconnect, it must be done while I/O is paused, as described in
Section 24.11, Scanning Storage Interconnects .

24 .10. Adding a St orage Device or Pat h

When adding a device, be aware that the path-based device name (/d ev/sd name, majo r: mi no r
number, and /d ev/d i sk/by-path name, for example) the system assigns to the new device may
have been previously in use by a device that has since been removed. As such, ensure that all old
references to the path-based device name have been removed. Otherwise, the new device may be
mistaken for the old device.

Pro ced u re 24 .11. Ad d a st o rag e d evice o r p at h

1. The first step in adding a storage device or path is to physically enable access to the new
storage device, or a new path to an existing device. This is done using vendor-specific
commands at the Fibre Channel or iSCSI storage server. When doing so, note the LUN value
for the new storage that will be presented to your host. If the storage server is Fibre Channel,
also take note of the World Wide Node Name (WWNN) of the storage server, and determine
whether there is a single WWNN for all ports on the storage server. If this is not the case, note
the World Wide Port Name (WWPN) for each port that will be used to access the new LUN.

2. Next, make the operating system aware of the new storage device, or path to an existing
device. The recommended command to use is:

$ echo "c t l" > /sys/class/scsi_host/hosth/scan

In the previous command, h is the HBA number, c is the channel on the HBA, t is the SCSI
target ID , and l is the LUN.

192
 Chapt er 2 4 . O nline St orage Management

Note

The older form of this command, echo "scsi ad d -si ng l e-d evi ce 0 0 0 0 " >
/pro c/scsi /scsi , is deprecated.

a. In some Fibre Channel hardware, a newly created LUN on the RAID array may not be
visible to the operating system until a Loop Initialization Protocol (LIP) operation is
performed. Refer to Section 24.11, Scanning Storage Interconnects for instructions
on how to do this.

Important

It will be necessary to stop I/O while this operation is executed if an LIP is

required.

b. If a new LUN has been added on the RAID array but is still not being configured by
the operating system, confirm the list of LUNs being exported by the array using the
sg _l uns command, part of the sg3_utils package. This will issue the SC SI R EP O R T
LUNS command to the RAID array and return a list of LUNs that are present.

For Fibre Channel storage servers that implement a single WWNN for all ports, you can
determine the correct h,c,and t values (i.e. HBA number, HBA channel, and SCSI target ID )
by searching for the WWNN in sysfs.

Examp le 24 .5. D et ermin co rrect h, c, an d t valu es

For example, if the WWNN of the storage server is 0 x50 0 6 0 16 0 9 0 20 3181, use:

$ grep 5006016090203181 /sys/class/fc_transport/*/node_name

This should display output similar to the following:

/sys/class/fc_transport/target5:0:2/node_name:0x5006016090203181
/sys/class/fc_transport/target5:0:3/node_name:0x5006016090203181
/sys/class/fc_transport/target6:0:2/node_name:0x5006016090203181
/sys/class/fc_transport/target6:0:3/node_name:0x5006016090203181

This indicates there are four Fibre Channel routes to this target (two single-channel HBAs,
each leading to two storage ports). Assuming a LUN value is 56 , then the following
command will configure the first path:

$ echo "0 2 56" > /sys/class/scsi_host/host5/scan

This must be done for each path to the new device.

For Fibre Channel storage servers that do not implement a single WWNN for all ports, you
can determine the correct HBA number, HBA channel, and SCSI target ID by searching for
each of the WWPNs in sysfs.

Another way to determine the HBA number, HBA channel, and SCSI target ID is to refer to

193
St orage Administ rat ion G uide

another device that is already configured on the same path as the new device. This can be
done with various commands, such as l sscsi , scsi _i d , mul ti path -l , and l s -l
/d ev/d i sk/by-*. This information, plus the LUN number of the new device, can be used as
shown above to probe and configure that path to the new device.

3. After adding all the SCSI paths to the device, execute the mul ti path command, and check to
see that the device has been properly configured. At this point, the device can be added to
md , LVM, mkfs, or mo unt, for example.

If the steps above are followed, then a device can safely be added to a running system. It is not
necessary to stop I/O to other devices while this is done. Other procedures involving a rescan (or a
reset) of the SCSI bus, which cause the operating system to update its state to reflect the current
device connectivity, are not recommended while storage I/O is in progress.

24 .11. Scanning St orage Int erconnect s

Certain commands allow you to reset, scan, or both reset and scan one or more interconnects, which
potentially adds and removes multiple devices in one operation. This type of scan can be disruptive,
as it can cause delays while I/O operations time out, and remove devices unexpectedly. Red Hat
recommends using interconnect scanning only when necessary. Observe the following restrictions
when scanning storage interconnects:

All I/O on the effected interconnects must be paused and flushed before executing the procedure,
and the results of the scan checked before I/O is resumed.

As with removing a device, interconnect scanning is not recommended when the system is under
memory pressure. To determine the level of memory pressure, run the vmstat 1 10 0 command.
Interconnect scanning is not recommended if free memory is less than 5% of the total memory in
more than 10 samples per 100. Also, interconnect scanning is not recommended if swapping is
active (non-zero si and so columns in the vmstat output). The free command can also display
the total memory.

The following commands can be used to scan storage interconnects:

echo "1" > /sys/cl ass/fc_ho st/ho st/i ssue_l i p

This operation performs a Loop Initialization Protocol (LIP), scans the interconnect, and
causes the SCSI layer to be updated to reflect the devices currently on the bus. Essentially,
an LIP is a bus reset, and causes device addition and removal. This procedure is
necessary to configure a new SCSI target on a Fibre Channel interconnect.

Note that i ssue_l i p is an asynchronous operation. The command can complete before
the entire scan has completed. You must monitor /var/l o g /messag es to determine when
i ssue_l i p finishes.

The l pfc, q l a2xxx, and bnx2fc drivers support i ssue_l i p. For more information
about the API capabilities supported by each driver in Red Hat Enterprise Linux, see
Table 24.1, Fibre Channel API Capabilities .

/usr/bi n/rescan-scsi -bus. sh

The /usr/bi n/rescan-scsi -bus. sh script was introduced in Red Hat

Enterprise Linux 5.4. By default, this script scans all the SCSI buses on the system, and
updates the SCSI layer to reflect new devices on the bus. The script provides additional
options to allow device removal, and the issuing of LIPs. For more information about this
script, including known issues, see Section 24.17, Adding/Removing a Logical Unit
Through rescan-scsi-bus.sh .

194
 Chapt er 2 4 . O nline St orage Management

echo "- - -" > /sys/cl ass/scsi _ho st/ho sth/scan

This is the same command as described in Section 24.10, Adding a Storage D evice or
Path to add a storage device or path. In this case, however, the channel number, SCSI
target ID , and LUN values are replaced by wildcards. Any combination of identifiers and
wildcards is allowed, so you can make the command as specific or broad as needed. This
procedure adds LUNs, but does not remove them.

mo d pro be --remo ve driver-name, mo d pro be driver-name

Running the mo d pro be --remo ve driver-name command followed by the mo d pro be

driver-name command completely re-initializes the state of all interconnects controlled by
the driver. D espite being rather extreme, using the described commands can be appropriate
in certain situations. The commands can be used, for example, to restart the driver with a
different module parameter value.

24 .12. iSCSI Discovery Configurat ion

The default iSCSI configuration file is /etc/i scsi /i scsi d . co nf. This file contains iSCSI
settings used by i scsi d and i scsi ad m.

D uring target discovery, the i scsi ad m tool uses the settings in /etc/i scsi /i scsi d . co nf to
create two types of records:

N o d e reco rd s in /var/l i b/i scsi /no d es

When logging into a target, i scsi ad m uses the settings in this file.

D isco very reco rd s in /var/l i b/i scsi /discovery_type

When performing discovery to the same destination, i scsi ad m uses the settings in this file.

Before using different settings for discovery, delete the current discovery records (i.e.
/var/l i b/i scsi /discovery_type) first. To do this, use the following command:

# iscsiadm -m discovery -t discovery_type -p target_IP:port -o delete [5]

Here, discovery_type can be either send targ ets, i sns, or fw.

For details on different types of discovery, refer to the DISCOVERY TYPES section of man i scsi ad m.

There are two ways to reconfigure discovery record settings:

Edit the /etc/i scsi /i scsi d . co nf file directly prior to performing a discovery. D iscovery
settings use the prefix d i sco very; to view them, run:

# iscsiadm -m discovery -t discovery_type -p target_IP:port

Alternatively, i scsi ad m can also be used to directly change discovery record settings, as in:

# iscsiadm -m discovery -t discovery_type -p target_IP:port -o update

-n setting -v %value

Refer to man i scsi ad m for more information on available settings and valid values for each.

195
St orage Administ rat ion G uide

After configuring discovery settings, any subsequent attempts to discover new targets will use the
new settings. Refer to Section 24.14, Scanning iSCSI Interconnects for details on how to scan for
new iSCSI targets.

For more information on configuring iSCSI target discovery, refer to the man pages of i scsi ad m and
i scsi d . The /etc/i scsi /i scsi d . co nf file also contains examples on proper configuration
syntax.

24 .13. Configuring iSCSI Offload and Int erface Binding

This chapter describes how to set up iSCSI interfaces in order to bind a session to a NIC port when
using software iSCSI. It also describes how to set up interfaces for use with network devices that
support offloading.

The network subsystem can be configured to determine the path/NIC that iSCSI interfaces should use
for binding. For example, if portals and NICs are set up on different subnets, then it is not necessary
to manually configure iSCSI interfaces for binding.

Before attempting to configure an iSCSI interface for binding, run the following command first:

$ ping -I ethX target_IP

If pi ng fails, then you will not be able to bind a session to a NIC. If this is the case, check the network
settings first.

24 .13.1. Viewing Available iface Configurat ions

iSCSI offload and interface binding is supported for the following iSCSI initiator implementations:

So f t ware iSC SI

This stack allocates an iSCSI host instance (that is, scsi _ho st) per session, with a single
connection per session. As a result, /sys/cl ass_scsi _ho st and /pro c/scsi will
report a scsi _ho st for each connection/session you are logged into.

O f f lo ad iSC SI

This stack allocates a scsi _ho st for each PCI device. As such, each port on a host bus
adapter will show up as a different PCI device, withh a different scsi _ho st per HBA port.

To manage both types of initiator implementations, i scsi ad m uses the i face structure. With this
structure, an i face configuration must be entered in /var/l i b/i scsi /i faces for each HBA port,
software iSCSI, or network device (ethX) used to bind sessions.

To view available i face configurations, run i scsi ad m -m i face. This will display i face
information in the following format:

iface_name
transport_name,hardware_address,ip_address,net_ifacename,initiator_name

Refer to the following table for an explanation of each value/setting.

T ab le 24 .2. if ace Set t in g s

196
 Chapt er 2 4 . O nline St orage Management

Set t in g D escrip t io n
i face_name i face configuration name.
transpo rt_name Name of driver
hard ware_ad d ress MAC address
i p_ad d ress IP address to use for this port
net_i face_name Name used for the vl an or alias binding of a
software iSCSI session. For iSCSI offloads,
net_i face_name will be <empty> because this
value is not persistent across reboots.
i ni ti ato r_name This setting is used to override a default name
for the initiator, which is defined in
/etc/i scsi /i ni ti ato rname. i scsi

Examp le 24 .6 . Samp le o u t p u t o f t h e i scsi ad m -m i face co mman d

The following is a sample output of the i scsi ad m -m i face command:

iface0 qla4xxx,00:c0:dd:08:63:e8,20.15.0.7,default,iqn.2005-
06.com.redhat:madmax
iface1 qla4xxx,00:c0:dd:08:63:ea,20.15.0.9,default,iqn.2005-
06.com.redhat:madmax

For software iSCSI, each i face configuration must have a unique name (with less than 65
characters). The i face_name for network devices that support offloading appears in the format
transport_name. hardware_name.

Examp le 24 .7. i scsi ad m -m i face o u t p u t wit h a C h elsio n et wo rk card

For example, the sample output of i scsi ad m -m i face on a system using a Chelsio network
card might appear as:

default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
cxgb3i.00:07:43:05:97:07 cxgb3i,00:07:43:05:97:07,<empty>,<empty>,
<empty>

It is also possible to display the settings of a specific i face configuration in a more friendly way. To
do so, use the option -I iface_name. This will display the settings in the following format:

iface.setting = value

Examp le 24 .8. U sin g i face set t in g s wit h a C h elsio co n verg ed n et wo rk ad ap t er

Using the previous example, the i face settings of the same Chelsio converged network adapter
(i.e. i scsi ad m -m i face -I cxg b3i . 0 0 : 0 7: 4 3: 0 5: 9 7: 0 7) would appear as:

# BEGIN RECORD 2.0-871

iface.iscsi_ifacename = cxgb3i.00:07:43:05:97:07
iface.net_ifacename = <empty>
iface.ipaddress = <empty>

197
St orage Administ rat ion G uide

iface.hwaddress = 00:07:43:05:97:07
iface.transport_name = cxgb3i
iface.initiatorname = <empty>
# END RECORD

24 .13.2. Configuring an iface for Soft ware iSCSI

As mentioned earlier, an i face configuration is required for each network object that will be used to
bind a session.

Before

To create an i face configuration for software iSCSI, run the following command:

# iscsiadm -m iface -I iface_name --op=new

This will create a new empty i face configuration with a specified iface_name. If an existing i face
configuration already has the same iface_name, then it will be overwritten with a new, empty one.

To configure a specific setting of an i face configuration, use the following command:

# iscsiadm -m iface -I iface_name --op=update -n iface.setting -v

hw_address

Examp le 24 .9 . Set MAC ad d ress o f i face0

For example, to set the MAC address (hard ware_ad d ress) of i face0 to
0 0 : 0 F: 1F: 9 2: 6 B: BF, run:

# iscsiadm -m iface -I iface0 --op=update -n iface.hwaddress -v

00:0F:1F:92:6B:BF

Warning

D o not use d efaul t or i ser as i face names. Both strings are special values used by
i scsi ad m for backward compatibility. Any manually-created i face configurations named
d efaul t or i ser will disable backwards compatibility.

24 .13.3. Configuring an iface for iSCSI Offload

By default i scsi ad m will create an i face configuration for each port. To view available i face
configurations, use the same command for doing so in software iSCSI, i.e. i scsi ad m -m i face.

Before using the i face of a network card for iSCSI offload, first set the IP address (target_IP [5])
that the device should use. For devices that use the be2i scsi driver, the IP address is configured in
the BIOS setup screen. For all other devices, to configure the IP address of the i face use:

# iscsiadm -m iface -I iface_name -o update -n iface.ipaddress -v

target_IP

198
 Chapt er 2 4 . O nline St orage Management

Examp le 24 .10. Set t h e i face IP ad d ress o f a C h elsio card

For example, to set the i face IP address to 20 . 15. 0 . 6 6 when using a card with the iface name
of cxgb3i.00:07:43:05:97:07, use:

# iscsiadm -m iface -I cxgb3i.00:07:43:05:97:07 -o update -n

iface.ipaddress -v 20.15.0.66

24 .13.4 . Binding/Unbinding an iface t o a Port al

Whenever i scsi ad m is used to scan for interconnects, it will first check the i face. transpo rt
settings of each i face configuration in /var/l i b/i scsi /i faces. The i scsi ad m utility will then
bind discovered portals to any i face whose i face. transpo rt is tcp.

This behavior was implemented for compatibility reasons. To override this, use the -I iface_name
to specify which portal to bind to an i face, as in:

# iscsiadm -m discovery -t st -p target_IP:port -I iface_name -P 1

[5]

By default, the i scsi ad m utility will not automatically bind any portals to i face configurations that
use offloading. This is because such i face configurations will not have i face. transpo rt set to
tcp. As such, the i face configurations need to be manually bound to discovered portals.

It is also possible to prevent a portal from binding to any existing i face. To do so, use d efaul t as
the iface_name, as in:

# iscsiadm -m discovery -t st -p IP:port -I default -P 1

To remove the binding between a target and i face, use:

# iscsiadm -m node -targetname proper_target_name -I iface0 --op=delete

[6]

To delete all bindings for a specific i face, use:

# iscsiadm -m node -I iface_name --op=delete

To delete bindings for a specific portal (e.g. for Equalogic targets), use:

# iscsiadm -m node -p IP:port -I iface_name --op=delete

199
St orage Administ rat ion G uide

Note

If there are no i face configurations defined in /var/l i b/i scsi /i face and the -I option
is not used, i scsi ad m will allow the network subsystem to decide which device a specific
portal should use.

24 .14 . Scanning iSCSI Int erconnect s

For iSCSI, if the targets send an iSCSI async event indicating new storage is added, then the scan is
done automatically.

However, if the targets do not send an iSCSI async event, you need to manually scan them using the
i scsi ad m utility. Before doing so, however, you need to first retrieve the proper --targ etname and
the --po rtal values. If your device model supports only a single logical unit and portal per target,
use i scsi ad m to issue a send targ ets command to the host, as in:

# iscsiadm -m discovery -t sendtargets -p target_IP:port

[5]

The output will appear in the following format:

target_IP:port,target_portal_group_tag proper_target_name

Examp le 24 .11. U sin g i scsi ad m t o issu e a send targ ets co mman d

For example, on a target with a proper_target_name of i q n. 19 9 2-

0 8. co m. netapp: sn. 336 15311 and a target_IP:port of 10 . 15. 85. 19 : 326 0 , the output
may appear as:

10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311
10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311

In this example, the target has two portals, each using target_ip:ports of
10 . 15. 84 . 19 : 326 0 and 10 . 15. 85. 19 : 326 0 .

To see which i face configuration will be used for each session, add the -P 1 option. This option
will print also session information in tree format, as in:

Target: proper_target_name
Portal: target_IP:port,target_portal_group_tag
Iface Name: iface_name

Examp le 24 .12. View i face co n f ig u rat io n

For example, with i scsi ad m -m d i sco very -t send targ ets -p 10 . 15. 85. 19 : 326 0 -
P 1, the output may appear as:

200
 Chapt er 2 4 . O nline St orage Management

Target: iqn.1992-08.com.netapp:sn.33615311
Portal: 10.15.84.19:3260,2
Iface Name: iface2
Portal: 10.15.85.19:3260,3
Iface Name: iface2

This means that the target i q n. 19 9 2-0 8. co m. netapp: sn. 336 15311 will use i face2 as its
i face configuration.

With some device models a single target may have multiple logical units and portals. In this case,
issue a send targ ets command to the host first to find new portals on the target. Then, rescan the
existing sessions using:

# iscsiadm -m session --rescan

You can also rescan a specific session by specifying the session's SID value, as in:

# iscsiadm -m session -r SID --rescan

[7]

If your device supports multiple targets, you will need to issue a send targ ets command to the hosts
to find new portals for each target. Rescan existing sessions to discover new logical units on existing
sessions using the --rescan option.

Important

The send targ ets command used to retrieve --targ etname and --po rtal values
overwrites the contents of the /var/l i b/i scsi /no d es database. This database will then
be repopulated using the settings in /etc/i scsi /i scsi d . co nf. However, this will not
occur if a session is currently logged in and in use.

To safely add new targets/portals or delete old ones, use the -o new or -o d el ete options,
respectively. For example, to add new targets/portals without overwriting
/var/l i b/i scsi /no d es, use the following command:

iscsiadm -m discovery -t st -p target_IP -o new

To delete /var/l i b/i scsi /no d es entries that the target did not display during discovery,
use:

iscsiadm -m discovery -t st -p target_IP -o delete

You can also perform both tasks simultaneously, as in:

iscsiadm -m discovery -t st -p target_IP -o delete -o new

The send targ ets command will yield the following output:

ip:port,target_portal_group_tag proper_target_name

201
St orage Administ rat ion G uide

Examp le 24 .13. O u t p u t o f t h e send targ ets co mman d

For example, given a device with a single target, logical unit, and portal, with eq ual l o g i c-
i scsi 1 as your target_name, the output should appear similar to the following:

10.16.41.155:3260,0 iqn.2001-05.com.equallogic:6-8a0900-ac3fe0101-
63aff113e344a4a2-dl585-03-1

Note that proper_target_name and ip:port,target_portal_group_tag are identical to the

values of the same name in Section 24.6.1, iSCSI API .

At this point, you now have the proper --targ etname and --po rtal values needed to manually
scan for iSCSI devices. To do so, run the following command:

# iscsiadm --mode node --targetname proper_target_name --portal

ip:port,target_portal_group_tag \ --login

[8]

Examp le 24 .14 . Fu ll i scsi ad m co mman d

Using our previous example (where proper_target_name is eq ual l o g i c-i scsi 1), the full
command would be:

# iscsiadm --mode node --targetname \ iqn.2001-05.com.equallogic:6-

8a0900-ac3fe0101-63aff113e344a4a2-dl585-03-1 \ --portal
10.16.41.155:3260,0 --login [8]

24 .15. Logging in t o an iSCSI T arget

As mentioned in Section 24.6, iSCSI , the iSCSI service must be running in order to discover or log
into targets. To start the iSCSI service, run:

# service iscsi start

When this command is executed, the iSCSI i ni t scripts will automatically log into targets where the
no d e. startup setting is configured as auto mati c. This is the default value of no d e. startup for
all targets.

To prevent automatic login to a target, set no d e. startup to manual . To do this, run the following
command:

# iscsiadm -m node --targetname proper_target_name -p target_IP:port -o

update -n node.startup -v manual

D eleting the entire record will also prevent automatic login. To do this, run:

# iscsiadm -m node --targetname proper_target_name -p target_IP:port -o

delete

202
 Chapt er 2 4 . O nline St orage Management

To automatically mount a file system from an iSCSI device on the network, add a partition entry for
the mount in /etc/fstab with the _netd ev option. For example, to automatically mount the iSCSI
device sd b to /mo unt/i scsi during startup, add the following line to /etc/fstab:

/dev/sdb /mnt/iscsi ext3 _netdev 0 0

To manually log in to an iSCSI target, use the following command:

# iscsiadm -m node --targetname proper_target_name -p target_IP:port -l

Note

The proper_target_name and target_IP:port refer to the full name and IP address/port
combination of a target. For more information, refer to Section 24.6.1, iSCSI API and
Section 24.14, Scanning iSCSI Interconnects .

24 .16. Resiz ing an Online Logical Unit

In most cases, fully resizing an online logical unit involves two things: resizing the logical unit itself
and reflecting the size change in the corresponding multipath device (if multipathing is enabled on
the system).

To resize the online logical unit, start by modifying the logical unit size through the array
management interface of your storage device. This procedure differs with each array; as such,
consult your storage array vendor documentation for more information on this.

Note

In order to resize an online file system, the file system must not reside on a partitioned device.

24 .16.1. Resiz ing Fibre Channel Logical Unit s

After modifying the online logical unit size, re-scan the logical unit to ensure that the system detects
the updated size. To do this for Fibre Channel logical units, use the following command:

$ echo 1 > /sys/block/sdX/device/rescan

Important

To re-scan Fibre Channel logical units on a system that uses multipathing, execute the
aforementioned command for each sd device (i.e. sd 1, sd 2, and so on) that represents a path
for the multipathed logical unit. To determine which devices are paths for a multipath logical
unit, use mul ti path -l l ; then, find the entry that matches the logical unit being resized. It is
advisable that you refer to the WWID of each entry to make it easier to find which one matches
the logical unit being resized.

203
St orage Administ rat ion G uide

24 .16.2. Resiz ing an iSCSI Logical Unit

After modifying the online logical unit size, re-scan the logical unit to ensure that the system detects
the updated size. To do this for iSCSI devices, use the following command:

# iscsiadm -m node --targetname target_name -R

[5]

Replace target_name with the name of the target where the device is located.

Note

You can also re-scan iSCSI logical units using the following command:

# iscsiadm -m node -R -I interface

Replace interface with the corresponding interface name of the resized logical unit (for
example, i face0 ). This command performs two operations:

It scans for new devices in the same way that the command echo "- - -" >
/sys/cl ass/scsi _ho st/host/scan does (refer to Section 24.14, Scanning iSCSI
Interconnects ).
It re-scans for new/modified logical units the same way that the command echo 1 >
/sys/bl o ck/sd X/d evi ce/rescan does. Note that this command is the same one used
for re-scanning Fibre Channel logical units.

24 .16.3. Updat ing t he Siz e of Your Mult ipat h Device

If multipathing is enabled on your system, you will also need to reflect the change in logical unit size
to the logical unit's corresponding multipath device (after resizing the logical unit). This can be done
through mul ti pathd . To do so, first ensure that mul ti pathd is running using servi ce
mul ti pathd status. Once you've verified that mul ti pathd is operational, run the following
command:

# multipathd -k"resize map multipath_device"

The multipath_device variable is the corresponding multipath entry of your device in

/d ev/mapper. D epending on how multipathing is set up on your system, multipath_device can
be either of two formats:

mpathX, where X is the corresponding entry of your device (for example, mpath0 )

a WWID ; for example, 36 0 0 50 8b4 0 0 10 5e210 0 0 0 9 0 0 0 0 0 4 9 0 0 0 0

To determine which multipath entry corresponds to your resized logical unit, run mul ti path -l l .
This displays a list of all existing multipath entries in the system, along with the major and minor
numbers of their corresponding devices.

204
 Chapt er 2 4 . O nline St orage Management

Important

D o not use mul ti pathd -k"resi ze map multipath_device" if there are any commands
queued to multipath_device. That is, do not use this command when the no _path_retry
parameter (in /etc/mul ti path. co nf) is set to "q ueue", and there are no active paths to
the device.

For more information about multipathing, refer to the Red Hat Enterprise Linux 7 DM Multipath guide.

24 .16.4 . Changing t he Read/Writ e St at e of an Online Logical Unit

Certain storage devices provide the user with the ability to change the state of the device from
Read/Write (R/W) to Read-Only (RO), and from RO to R/W. This is typically done through a
management interface on the storage device. The operating system will not automatically update its
view of the state of the device when a change is made. Follow the procedures described in this
chapter to make the operating system aware of the change.

Run the following command, replacing XYZ with the desired device designator, to determine the
operating system's current view of the R/W state of a device:

# bl o ckd ev --g etro /d ev/sd XYZ

The following command is also available for Red Hat Enterprise Linux 7:

# cat /sys/bl o ck/sd XYZ/ro 1 = read -o nl y 0 = read -wri te

When using multipath, refer to the ro or rw field in the second line of output from the mul ti path -l l
command. For example:

36001438005deb4710000500000640000 dm-8 GZ,GZ500

[size=20G][features=0][hwhandler=0][ro]
\_ round-robin 0 [prio=200][active]
\_ 6:0:4:1 sdax 67:16 [active][ready]
\_ 6:0:5:1 sday 67:32 [active][ready]
\_ round-robin 0 [prio=40][enabled]
\_ 6:0:6:1 sdaz 67:48 [active][ready]
\_ 6:0:7:1 sdba 67:64 [active][ready]

To change the R/W state, use the following procedure:

Pro ced u re 24 .12. C h an g e t h e R /W st at e

1. To move the device from RO to R/W, see step 2.

To move the device from R/W to RO, ensure no further writes will be issued. D o this by
stopping the application, or through the use of an appropriate, application-specific action.

Ensure that all outstanding write I/Os are complete with the following command:

# bl o ckd ev --fl ushbufs /d ev/device

Replace device with the desired designator; for a device mapper multipath, this is the entry for
your device in d ev/mapper. For example, /d ev/mapper/mpath3.

205
St orage Administ rat ion G uide

2. Use the management interface of the storage device to change the state of the logical unit
from R/W to RO, or from RO to R/W. The procedure for this differs with each array. Consult
applicable storage array vendor documentation for more information.

3. Perform a re-scan of the device to update the operating system's view of the R/W state of the
device. If using a device mapper multipath, perform this re-scan for each path to the device
before issuing the command telling multipath to reload its device maps.

This process is explained in further detail in Section 24.16.4.1, Rescanning logical units .

2 4 .1 6 .4 .1 . Re scanning lo gical unit s

After modifying the online logical unit Read/Write state, as described in Section 24.16.4, Changing
the Read/Write State of an Online Logical Unit , re-scan the logical unit to ensure the system detects
the updated state with the following command:

# echo 1 > /sys/bl o ck/sd X/d evi ce/rescan

To re-scan logical units on a system that uses multipathing, execute the above command for each sd
device that represents a path for the multipathed logical unit. For example, run the command on sd1,
sd2 and all other sd devices. To determine which devices are paths for a multipath unit, use
mul ti path -11, then find the entry that matches the logical unit to be changed.

Examp le 24 .15. U se o f t h e mul ti path -11 co mman d

For example, the mul ti path -11 above shows the path for the LUN with WWID
36001438005deb4710000500000640000. In this case, enter:

# echo 1 > /sys/bl o ck/sd ax/d evi ce/rescan

# echo 1 > /sys/bl o ck/sd ay/d evi ce/rescan
# echo 1 > /sys/bl o ck/sd az/d evi ce/rescan
# echo 1 > /sys/bl o ck/sd ba/d evi ce/rescan

2 4 .1 6 .4 .2 . Updat ing t he R/W st at e o f a m ult ipat h de vice

If multipathing is enabled, after rescanning the logical unit, the change in its state will need to be
reflected in the logical unit's corresponding multipath drive. D o this by reloading the multipath device
maps with the following command:

# mul ti path -r

The mul ti path -11 command can then be used to confirm the change.

2 4 .1 6 .4 .3. Do cum e nt at io n

Further information can be found in the Red Hat Knowledgebase. To access this, navigate to
https://www.redhat.com/wapps/sso/login.html?redirect=https://access.redhat.com/knowledge/ and log
in. Then access the article at https://access.redhat.com/kb/docs/D OC-32850.

24 .17. Adding/Removing a Logical Unit T hrough rescan-scsi-bus.sh

206
 Chapt er 2 4 . O nline St orage Management

The sg 3_uti l s package provides the rescan-scsi -bus. sh script, which can automatically
update the logical unit configuration of the host as needed (after a device has been added to the
system). The rescan-scsi -bus. sh script can also perform an i ssue_l i p on supported devices.
For more information about how to use this script, refer to rescan-scsi -bus. sh --hel p.

To install the sg 3_uti l s package, run yum i nstal l sg 3_uti l s.

Known Issues Wit h rescan-scsi-bus.sh

When using the rescan-scsi -bus. sh script, take note of the following known issues:

In order for rescan-scsi -bus. sh to work properly, LUN0 must be the first mapped logical unit.
The rescan-scsi -bus. sh can only detect the first mapped logical unit if it is LUN0 . The
rescan-scsi -bus. sh will not be able to scan any other logical unit unless it detects the first
mapped logical unit even if you use the --no o ptscan option.

A race condition requires that rescan-scsi -bus. sh be run twice if logical units are mapped for
the first time. D uring the first scan, rescan-scsi -bus. sh only adds LUN0 ; all other logical units
are added in the second scan.

A bug in the rescan-scsi -bus. sh script incorrectly executes the functionality for recognizing a
change in logical unit size when the --remo ve option is used.

The rescan-scsi -bus. sh script does not recognize ISCSI logical unit removals.

24 .18. Modifying Link Loss Behavior

This section describes how to modify the link loss behavior of devices that use either Fibre Channel
or iSCSI protocols.

24 .18.1. Fibre Channel

If a driver implements the Transport d ev_l o ss_tmo callback, access attempts to a device through a
link will be blocked when a transport problem is detected. To verify if a device is blocked, run the
following command:

$ cat /sys/block/device/device/state

This command will return bl o cked if the device is blocked. If the device is operating normally, this
command will return runni ng .

Pro ced u re 24 .13. D et ermin in g T h e St at e o f a R emo t e Po rt

1. To determine the state of a remote port, run the following command:

$ cat
/sys/class/fc_remote_port/rport-H:B:R/port_state

2. This command will return Bl o cked when the remote port (along with devices accessed
through it) are blocked. If the remote port is operating normally, the command will return
O nl i ne.

3. If the problem is not resolved within d ev_l o ss_tmo seconds, the rport and devices will be
unblocked and all I/O running on that device (along with any new I/O sent to that device) will
be failed.

207
St orage Administ rat ion G uide

Pro ced u re 24 .14 . C h an g in g d ev_l o ss_tmo

To change the d ev_l o ss_tmo value, echo in the desired value to the file. For example, to set
d ev_l o ss_tmo to 30 seconds, run:

$ echo 30 >
/sys/class/fc_remote_port/rport-H:B:R/dev_loss_tmo

For more information about d ev_l o ss_tmo , refer to Section 24.3.1, Fibre Channel API .

When a link loss exceeds d ev_l o ss_tmo , the scsi _d evi ce and sd N devices are removed.
Typically, the Fibre Channel class will leave the device as is; i.e. /d ev/sd x will remain /d ev/sd x.
This is because the target binding is saved by the Fibre Channel driver so when the target port
returns, the SCSI addresses are recreated faithfully. However, this cannot be guaranteed; the sd x will
be restored only if no additional change on in-storage box configuration of LUNs is made.

24 .18.2. iSCSI Set t ings Wit h d m-mul ti path

If d m-mul ti path is implemented, it is advisable to set iSCSI timers to immediately defer commands
to the multipath layer. To configure this, nest the following line under d evi ce { in
/etc/mul ti path. co nf:

features "1 queue_if_no_path"

This ensures that I/O errors are retried and queued if all paths are failed in the d m-mul ti path layer.

You may need to adjust iSCSI timers further to better monitor your SAN for problems. Available iSCSI
timers you can configure are NOP-Out Interval/Timeouts and repl acement_ti meo ut, which are
discussed in the following sections.

2 4 .1 8 .2 .1 . NOP-Out Int e rval/T im e o ut

To help monitor problems the SAN, the iSCSI layer sends a NOP-Out request to each target. If a NOP-
Out request times out, the iSCSI layer responds by failing any running commands and instructing the
SCSI layer to requeue those commands when possible.

When d m-mul ti path is being used, the SCSI layer will fail those running commands and defer them
to the multipath layer. The multipath layer then retries those commands on another path. If d m-
mul ti path is not being used, those commands are retried five times before failing altogether.

Intervals between NOP-Out requests are 10 seconds by default. To adjust this, open
/etc/i scsi /i scsi d . co nf and edit the following line:

node.conn[0].timeo.noop_out_interval = [interval value]

Once set, the iSCSI layer will send a NOP-Out request to each target every [interval value] seconds.

By default, NOP-Out requests time out in 10 seconds [9 ] . To adjust this, open

/etc/i scsi /i scsi d . co nf and edit the following line:

node.conn[0].timeo.noop_out_timeout = [timeout value]

This sets the iSCSI layer to timeout a NOP-Out request after [timeout value] seconds.

SC SI Erro r H an d ler

208
 Chapt er 2 4 . O nline St orage Management

If the SCSI Error Handler is running, running commands on a path will not be failed immediately
when a NOP-Out request times out on that path. Instead, those commands will be failed after
repl acement_ti meo ut seconds. For more information about repl acement_ti meo ut, refer to
Section 24.18.2.2, repl acement_ti meo ut .

To verify if the SCSI Error Handler is running, run:

# iscsiadm -m session -P 3

2 4 .1 8 .2 .2 . repl acement_ti meo ut

repl acement_ti meo ut controls how long the iSCSI layer should wait for a timed-out path/session
to reestablish itself before failing any commands on it. The default repl acement_ti meo ut value is
120 seconds.

To adjust repl acement_ti meo ut, open /etc/i scsi /i scsi d . co nf and edit the following line:

node.session.timeo.replacement_timeout = [replacement_timeout]

The 1 q ueue_i f_no _path option in /etc/mul ti path. co nf sets iSCSI timers to immediately
defer commands to the multipath layer (refer to Section 24.18.2, iSCSI Settings With d m-
mul ti path ). This setting prevents I/O errors from propagating to the application; because of this,
you can set repl acement_ti meo ut to 15-20 seconds.

By configuring a lower repl acement_ti meo ut, I/O is quickly sent to a new path and executed (in
the event of a NOP-Out timeout) while the iSCSI layer attempts to re-establish the failed path/session.
If all paths time out, then the multipath and device mapper layer will internally queue I/O based on the
settings in /etc/mul ti path. co nf instead of /etc/i scsi /i scsi d . co nf.

Important

Whether your considerations are failover speed or security, the recommended value for
repl acement_ti meo ut will depend on other factors. These factors include the network,
target, and system workload. As such, it is recommended that you thoroughly test any new
configurations to repl acements_ti meo ut before applying it to a mission-critical system.

24 .18.3. iSCSI Root

When accessing the root partition directly through an iSCSI disk, the iSCSI timers should be set so
that iSCSI layer has several chances to try to reestablish a path/session. In addition, commands
should not be quickly re-queued to the SCSI layer. This is the opposite of what should be done when
d m-mul ti path is implemented.

To start with, NOP-Outs should be disabled. You can do this by setting both NOP-Out interval and
timeout to zero. To set this, open /etc/i scsi /i scsi d . co nf and edit as follows:

node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0

In line with this, repl acement_ti meo ut should be set to a high number. This will instruct the
system to wait a long time for a path/session to reestablish itself. To adjust repl acement_ti meo ut,
open /etc/i scsi /i scsi d . co nf and edit the following line:

209
St orage Administ rat ion G uide

node.session.timeo.replacement_timeout = replacement_timeout

After configuring /etc/i scsi /i scsi d . co nf, you must perform a re-discovery of the affected
storage. This will allow the system to load and use any new values in /etc/i scsi /i scsi d . co nf.
For more information on how to discover iSCSI devices, refer to Section 24.14, Scanning iSCSI
Interconnects .

Co nfiguring T im e o ut s fo r a Spe cific Se ssio n

You can also configure timeouts for a specific session and make them non-persistent (instead of
using /etc/i scsi /i scsi d . co nf). To do so, run the following command (replace the variables
accordingly):

# iscsiadm -m node -T target_name -p target_IP:port -o update -n

node.session.timeo.replacement_timeout -v $timeout_value

Important

The configuration described here is recommended for iSCSI sessions involving root partition
access. For iSCSI sessions involving access to other types of storage (namely, in systems that
use d m-mul ti path), refer to Section 24.18.2, iSCSI Settings With d m-mul ti path .

24 .19. Cont rolling t he SCSI Command T imer and Device St at us

The Linux SCSI layer sets a timer on each command. When this timer expires, the SCSI layer will
quiesce the host bus adapter (HBA) and wait for all outstanding commands to either time out or
complete. Afterwards, the SCSI layer will activate the driver's error handler.

When the error handler is triggered, it attempts the following operations in order (until one
successfully executes):

1. Abort the command.

2. Reset the device.

3. Reset the bus.

4. Reset the host.

If all of these operations fail, the device will be set to the o ffl i ne state. When this occurs, all I/O to
that device will be failed, until the problem is corrected and the user sets the device to runni ng .

The process is different, however, if a device uses the Fibre Channel protocol and the rpo rt is
blocked. In such cases, the drivers wait for several seconds for the rpo rt to become online again
before activating the error handler. This prevents devices from becoming offline due to temporary
transport problems.

Device St at es

To display the state of a device, use:

$ cat /sys/block/device-name/device/state

210
 Chapt er 2 4 . O nline St orage Management

To set a device to the runni ng state, use:

# echo running > /sys/block/device-name/device/state

Command T imer

To control the command timer, modify the /sys/bl o ck/device-name/d evi ce/ti meo ut file:

# echo value > /sys/block/device-name/device/timeout

Replace value in the command with the timeout value, in seconds, that you want to implement.

24 .20. Online St orage Configurat ion T roubleshoot ing

This section provides solution to common problems users experience during online storage
reconfiguration.

Lo g ical u n it remo val st at u s is n o t ref lect ed o n t h e h o st .

When a logical unit is deleted on a configured filer, the change is not reflected on the host.
In such cases, l vm commands will hang indefinitely when d m-mul ti path is used, as the
logical unit has now become stale.

To work around this, perform the following procedure:

Pro ced u re 24 .15. Wo rkin g Aro u n d St ale Lo g ical U n it s

1. D etermine which mpath link entries in /etc/l vm/cache/. cache are specific to
the stale logical unit. To do this, run the following command:

$ ls -l /dev/mpath | grep stale-logical-unit

Examp le 24 .16 . D et ermin e sp ecif ic mpath lin k en t ries

For example, if stale-logical-unit is

3600d0230003414f30000203a7bc41a00, the following results may appear:

lrwxrwxrwx 1 root root 7 Aug 2 10:33

/3600d0230003414f30000203a7bc41a00 -> ../dm-4
lrwxrwxrwx 1 root root 7 Aug 2 10:33
/3600d0230003414f30000203a7bc41a00p1 -> ../dm-5

This means that 3600d0230003414f30000203a7bc41a00 is mapped to two

mpath links: d m-4 and d m-5.

2. Next, open /etc/l vm/cache/. cache. D elete all lines containing stale-
logical-unit and the mpath links that stale-logical-unit maps to.

Examp le 24 .17. D elet e relevan t lin es

Using the same example in the previous step, the lines you need to delete are:

211
St orage Administ rat ion G uide

/dev/dm-4
/dev/dm-5
/dev/mapper/3600d0230003414f30000203a7bc41a00
/dev/mapper/3600d0230003414f30000203a7bc41a00p1
/dev/mpath/3600d0230003414f30000203a7bc41a00
/dev/mpath/3600d0230003414f30000203a7bc41a00p1

24 .21. Configuring Maximum T ime for Error Recovery wit h eh_deadline

Important

In most scenarios, you do not need to enable the eh_deadline parameter. Using the
eh_deadline parameter can be useful in certain specific scenarios, for example if a link loss
occurs between a Fibre Channel switch and a target port, and the Host Bus Adapter (HBA)
does not receive Registered State Change Notifications (RSCNs). In such a case, I/O requests
and error recovery commands all time out rather than encounter an error. Setting
eh_deadline in this environment puts an upper limit on the recovery time, which enables the
failed I/O to be retried on another available path by multipath.

However, if RSCNs are enabled, the HBA does not register the link becoming unavailable, or
both, the eh_deadline functionality provides no additional benefit, as the I/O and error
recovery commands fail immediately, which allows multipath to retry.

The SCSI host object eh_deadline parameter enables you to configure the maximum amount of
time that the SCSI error handling mechanism attempts to perform error recovery before stopping and
resetting the entire HBA.

The value of the eh_deadline is specified in seconds. The default setting is o ff, which disables the
time limit and allows all of the error recovery to take place. In addition to using sysfs, a default value
can be set for all SCSI HBAs by using the scsi_mod.eh_deadline kernel parameter.

Note that when eh_deadline expires, the HBA is reset, which affects all target paths on that HBA,
not only the failing one. As a consequence, I/O errors can occur if some of the redundant paths are
not available for other reasons. Enable eh_deadline only if you have a fully redundant multipath
configuration on all targets.

[5] The target_IP and port variab les refer to the IP ad d res s and p o rt c o mb inatio n o f a targ et/p o rtal,
res p ec tively. Fo r mo re info rmatio n, refer to Sec tio n 24.6 .1, iSCSI API and Sec tio n 24.14, Sc anning
iSCSI Interc o nnec ts .

[6 ] Refer to Sec tio n 24.14, Sc anning iSCSI Interc o nnec ts fo r info rmatio n o n proper_target_name .

[7] Fo r info rmatio n o n ho w to retrieve a s es s io n' s SID value, refer to Sec tio n 24.6 .1, iSCSI API .

[8 ] This is a s ing le c o mmand s p lit into multip le lines , to ac c o mmo d ate p rinted and PDF vers io ns o f this
d o c ument. All c o nc atenated lines p rec ed ed b y the b ac ks las h (\) s ho uld b e treated as o ne
c o mmand , s ans b ac ks las hes .

[9 ] Prio r to Red Hat Enterp ris e Linux 5.4, the d efault NO P-O ut req ues ts time o ut was 15 s ec o nd s .

212
 Chapt er 2 5. Device Mapper Mult ipat hing and Virt ual St orage

Chapter 25. Device Mapper Multipathing and Virtual Storage

Red Hat Enterprise Linux 7 supports DM-Multipath and virtual storage. Both features are documented in
detail in the Red Hat books DM Multipath and Virtualization Deployment and Administration Guide.

25.1. Virt ual St orage

Red Hat Enterprise Linux 7 supports the following file systems/online storage methods for virtual
storage:

Fibre Channel

iSCSI

NFS

GFS2

Virtualization in Red Hat Enterprise Linux 7 uses l i bvi rt to manage virtual instances. The
l i bvi rt utility uses the concept of storage pools to manage storage for virtualized guests. A storage
pool is storage that can be divided up into smaller volumes or allocated directly to a guest. Volumes
of a storage pool can be allocated to virtualized guests. There are two categories of storage pools
available:
Lo cal st o rag e p o o ls

Local storage covers storage devices, files or directories directly attached to a host. Local
storage includes local directories, directly attached disks, and LVM Volume Groups.

N et wo rked ( sh ared ) st o rag e p o o ls

Networked storage covers storage devices shared over a network using standard
protocols. It includes shared storage devices using Fibre Channel, iSCSI, NFS, GFS2, and
SCSI RD MA protocols, and is a requirement for migrating guest virtualized guests between
hosts.

Important

For comprehensive information on the deployment and configuration of virtual storage

instances in your environment, refer to the Virtualization Deployment and Administration Guide
provided by Red Hat.

25.2. DM-Mult ipat h

D evice Mapper Multipathing (D M-Multipath) is a feature that allows you to configure multiple I/O
paths between server nodes and storage arrays into a single device. These I/O paths are physical
SAN connections that can include separate cables, switches, and controllers. Multipathing
aggregates the I/O paths, creating a new device that consists of the aggregated paths.

D M-Multipath are used primarily for the following reasons:

R ed u n d an cy

D M-Multipath can provide failover in an active/passive configuration. In an active/passive

configuration, only half the paths are used at any time for I/O. If any element of an I/O path

213
St orage Administ rat ion G uide

(the cable, switch, or controller) fails, D M-Multipath switches to an alternate path.

Imp ro ved Perf o rman ce

D M-Multipath can be configured in active/active mode, where I/O is spread over the paths in
a round-robin fashion. In some configurations, D M-Multipath can detect loading on the I/O
paths and dynamically re-balance the load.

Important

For comprehensive information on the deployment and configuration of D M-Multipath in your

environment, refer to the Using DM-Multipath guide provided by Red Hat.

214
 Chapt er 2 6 . Ext ernal Array Management (libSt orageMgmt )

Chapter 26. External Array Management (l i bSto rag eMg mt)

Red Hat Enterprise Linux 7 ships with a new external array management package called
l i bSto rag eMg mt.

26.1. What is l i bSto rag eMg mt

The l i bSto rag eMg mt package is a storage array independent Application Programming Interface
(API). It provides a stable and consistent API that allows developers the ability to programmatically
manage different storage arrays and leverage the hardware accelerated features provided.

This library is used as a building block for other higher level management tools and applications.
End system administrators can also use it as a tool to manually manage storage and automate
storage management tasks with the use of scripts.

The l i bSto rag eMg mt package allows operations such as:

List storage pools, volumes, access groups, or file systems.

Create and delete volumes, access groups, file systems, or NFS exports.

Grant and remove access to volumes, access groups, or initiators.

Replicate volumes with snapshots, clones, and copies.

Create and delete access groups and edit members of a group.

Server resources such as CPU and interconnect bandwidth are not utilized because the operations
are all done on the array.

The package provides:

A stable C and Python API for client application and plug-in developers.

A commandl line interface that utilizes the library (l smcl i ).

A daemon that executes the plug-in (l smd ).

A simulator plug-in that allows the testing of client applications (si m).

Plug-in architecture for interfacing with arrays.

Warning

This library and its associated tool have the ability to destroy any and all data located on the
arrays it manages. It is highly recommended to develop and test applications and scripts
against the storage simulator pulug-in to remove any logic errors before working with
production systems. Testing applications and scripts on actual non-production hardware
before deploying to production is also strongly encouraged if possible.

The l i bSto rag eMg mt package in Red Hat Enterprise Linux 7 adds a default udev rule to handle the
REPORTED LUNS D ATA HAS CHANGED unit attention.

When a storage configuration change has taken place, one of several Unit Attention ASC/ASCQ
codes reports the change. A uevent is then generated and is rescanned automatically with sysfs.

215
St orage Administ rat ion G uide

The file /l i b/ud ev/rul es. d /9 0 -scsi -ua. rul es contains example rules to enumerate other
events that the kernel can generate.

The l i bSto rag eMg mt library uses a plug-in architecture to accommodate differences in storage
arrays. For more information on l i bSto rag eMg mt plug-ins and how to write them, refer to Red Hat's
Developer Guide.

26.2. T erminology from l i bSto rag eMg mt

D ifferent array vendors and storage standards use different terminology to refer to similar
functionality. This library uses the following terminology.

St o rag e array

Any storage system that provides block access (FC, FCoE, iSCSI) or file access through
Network Attached Storage (NAS).

Vo lu me

Storage Area Network (SAN) Storage Arrays can expose a volume to the Host Bus Adapter
(HBA) over different transports, such as FC, iSCSI, or FCoE. The host OS treats it as block
devices. One volume can be exposed to many disks if mul ti path[2] is enabled).

This is also known as the Logical Unit Number (LUN), StorageVolume with SNIA
terminology, or virtual disk.

Po o l

A group of storage spaces. File systems or volumes can be created from a pool. Pools can
be created from disks, volumes, and other pools. A pool may also hold RAID settings or thin
provisioning settings.

This is also known as a StoragePool with SNIA Terminology.

Sn ap sh o t

A point in time, read only, space efficent copy of data.

This is also known as a read only snapshot.

C lo n e

A point in time, read writeable, space efficent copy of data.

This is also known as a read writeable snapshot.

Copy

A full bitwise copy of the data. It occupies the full space.

Mirro r

A continuously updated copy (synchronous and asynchronous).

Access g ro u p

Collections of iSCSI, FC, and FCoE initiators which are granted access to one or more
storage volumes. This ensures that only storage volumes are accessibly by the specified
initiators.

216
 Chapt er 2 6 . Ext ernal Array Management (libSt orageMgmt )

This is also known as an initiator group.

Access G ran t

Exposing a volume to a specified access group or initiator. The l i bSto rag eMg mt library
currently does not support LUN mapping with the ability to choose a specific logical unit
number. The l i bSto rag eMg mt library allows the storage array to select the next available
LUN for assignment. If configuring a boot from SAN or masking more than 256 volumes be
sure to read the OS, Storage Array, or HBA documents.

Access grant is also known as LUN Masking.

Syst em

Represents a storage array or a direct attached storage RAID .

File syst em

A Network Attached Storage (NAS) storage array can expose a file system to host an OS
through an IP network, using either NFS or CIFS protocol. The host OS treats it as a mount
point or a folder containing files depending on the client operating system.

D isk

The physical disk holding the data. This is normally used when creating a pool with RAID
settings.

This is also known as a D iskD rive using SNIA Terminology.

In it iat o r

In Fibre Channel (FC) or Fibre Channel over Ethernet (FCoE), the intiator is the World Wide
Port Name (WWPN) or World Wide Node Name (WWNN). In iSCSI, the initiator is the iSCSI
Qualified Name (IQN). In NFS or CIFS, the initiator is the host name or the IP address of the
host.

C h ild d ep en d en cy

Some arrays have an implicit relationship between the origin (parent volume or file system)
and the child (such as a snapshot or a clone). For example, it is impossible to delete the
parent if it has one or more depend children. The API provides methods to determine if any
such relationship exists and a method to remove the dependency by replicating the
required blocks.

26.3. Inst allat ion

To install l i bSto rag eMg mt for use of the command line, required run-time libraries and simulator
plug-ins use the following command:

$ sudo yum install libstoragemgmt libstoragemgmt-python

To develop C applications that utilize the library, install the l i bsto rag emg mt-d evel and,
optionally, the l i bsto rag e-d ebug i nfo packages with the following command:

$ sudo yum install libstoragemgmt-devel libstoragemgmt-debuginfo

217
St orage Administ rat ion G uide

To install l i bSto rag eMg mt for use with hardware arrays, select one or more of the appropriate
plug-in packages with the following command:

$ sudo yum install libstoragemgmt-name-plugin

The following plug-ins that are available include:

lib st o rag emg mt - smis- p lu g in

Generic SMI-S array support.

lib st o rag emg mt - n et ap p - p lu g in

Specific support for NetApp files.

lib st o rag emg mt - n st o r- p lu g in

Specific support for NexentaStor.

lib st o rag emg mt - t arg et d - p lu g in

Specific support for targetd.

The daemon is then installed and configured to run at start up but will not do so until the next reboot.
To use it immediately without rebooting, start the daemon manually.

To manage an array requires support through a plug-in. The base install package includes open
source plug-ins for a number of different vendors. Additional plug-in packages will be available
separately as array support improves. Currently supported hardware is constantly changing and
improving.

The libStorageMgmt daemon (l smd ) behaves like any standard service for the system.

To check on the status of libStorageMgmt use:

$ sudo systemctl status libstoragemgmt

To stop the service use:

$ sudo systemctl stop libstoragemgmt

To start the service use:

$ sudo systemctl start libstoragemgmt

26.4 . Use of t he l i bSto rag eMg mt

To use l i bSto rag eMg mt interactively, use the l smcl i command.

The l smcl i tool requires two things to run:

A Uniform Resource Identifier (URI) which is used to identify the plug-in to connect to the array
and any configurable options the array requires.

A valid username and password for the array.

URI has the following form:

218
 Chapt er 2 6 . Ext ernal Array Management (libSt orageMgmt )

plugin+optional-transport://username@host:port/?query-string-parameters

Each plug-in has different requirements for what is needed.

Examp le 26 .1. Examp les o f d if f eren t p lu g - in req u iremen t s

Simu lat o r p lu g - in t h at req u ires n o u ser n ame o r p asswo rd

si m: //

N et Ap p p lu g - in o ver SSL wit h u ser n ame ro o t

o ntap+ ssl : //ro o t@ filer. company. co m/

SMI- S p lu g - in o ver SSL f o r EMC array

smi s+ ssl : //ad mi n@ provider. co m: 59 89 /?namespace= ro o t/emc

There are three options to use the URI:

1. Pass the URI as part of the command.

$ lsmcli -u sim://...

2. Store the URI in an environmental vairable.

$ export LSMCLI_URI=sim:// & & lsmcli ...

3. Place the URI in the file ~ /. l smcl i , which contains name-value pairs separated by " =" . The
only currently supported configuration is 'uri'.

D etermining which URI to use needs to be done in this order. If all three are supplied, only the first
one on the command line will be used.

Supply the password by specifying -P on the command line or by placing it in an environmental

variable LSMCLI_PASSWORD .

Examp le 26 .2. Examp les o f l smcl i

An example for using the command line to create a new volume and making it visible to an
initiator.

List arrays that are serviced by this connection.

$ lsmcli list --type SYSTEMS

ID | Name | Status
-------+-------------------------------+--------
sim-01 | LSM simulated storage plug-in | OK

List storage pools.

$ lsmcli list --type POOLS -H

ID | Name | Total space | Free space |
System ID

219
St orage Administ rat ion G uide

-----+---------------+----------------------+----------------------+--
---------
POO2 | Pool 2 | 18446744073709551616 | 18446744073709551616 |
sim-01
POO3 | Pool 3 | 18446744073709551616 | 18446744073709551616 |
sim-01
POO1 | Pool 1 | 18446744073709551616 | 18446744073709551616 |
sim-01
POO4 | lsm_test_aggr | 18446744073709551616 | 18446744073709551616 |
sim-01

Create a volume.

$ lsmcli volume-create --name volume_name --size 20G --pool POO1 -H

ID | Name | vpd83 | bs | #blocks
| status | ...
-----+-------------+----------------------------------+-----+---------
-+--------+----
Vol1 | volume_name | F7DDF7CA945C66238F593BC38137BD2F | 512 | 41943040
| OK | ...

Create an access group with an iSCSI initiator in it.

$ lsmcli --create-access-group example_ag --id iqn.1994-

05.com.domain:01.89bd01 --type ISCSI --system sim-01
ID | Name | Initiator ID
|SystemID
---------------------------------+------------+-----------------------
-----------+--------
782d00c8ac63819d6cca7069282e03a0 | example_ag | iqn.1994-
05.com.domain:01.89bd01 |sim-01

Create an access group with an iSCSI intiator in it:

$ lsmcli access-group-create --name example_ag --init iqn.1994-

05.com.domain:01.89bd01 --init-type ISCSI --sys sim-01
ID | Name | Initiator IDs
| System ID
---------------------------------+------------+-----------------------
-----------+-----------
782d00c8ac63819d6cca7069282e03a0 | example_ag | iqn.1994-
05.com.domain:01.89bd01 | sim-01

Allow the access group visibility to the newly created volume:

$ lsmcli access-group-grant --ag 782d00c8ac63819d6cca7069282e03a0 --vol

Vol1 --access RW

The design of the library provides for a process separation between the client and the plug-in by
means of inter-process communication (IPC). This prevents bugs in the plug-in from crashing the client
application. It also provides a means for plug-in writers to write plug-ins with a license of their own
choosing. When a client opens the library passing a URI, the client library looks at the URI to
determine which plug-in should be used.

220
 Chapt er 2 6 . Ext ernal Array Management (libSt orageMgmt )

The plug-ins are technically stand alone applications but they are designed to have a file descriptor
passed to them on the command line. The client library then opens the appropriate Unix domain
socket which causes the daemon to fork and execute the plug-in. This gives the client library a point
to point communcation channel with the plug-in. The daemon can be restarted without affecting
existing clients. While the client has the library open for that plug-in, the plug-in process is running.
After one or more commands are sent and the plug-in is closed, the plug-in process cleans up and
then exits.

The default behavior of lsmcli is to wait until the operation is completee. D epending on the requested
operations, this could potentially could take many hours. To allow a return to normal usage, it is
possible to use the -b option on the command line. If the exit code is 0 the command is completed. If
the exit code is 7 the command is in progress and a job identifier is written to standard output. The
user or script can then take the job ID and query the status of the command as needed by using
l smcl i --jo bstatus JobID. If the job is now completed, the exit value will be 0 and the results
printed to standard output. If the command is still in progress, the return value will be 7 and the
percentage complete will be printed to the standard output.

Examp le 26 .3. An asyn ch ro n o u s examp le

Create a volume passing the -b option so that the command returns immediately.

$ lsmcli volume-create --name async_created --size 20G --pool POO1 -b

JOB_3

Check to see what the exit value was, remembering that 7 indicates the job is still in progress.

$ echo $?
7

Check to see if the job is completed.

$ lsmcli job-status --job JOB_3

33

Check to see what the exit value was, remembering that 7 indicates the job is still in progress so
the standard output is the percentage done or 33% based on the above screen.

$ echo $?
7

Wait some more and check it again, remembering that exit 0 means success and standard out
displays the new volume.

$ lsmcli job-status --job JOB_3

ID | Name | vpd83 | Block Size
| ...
-----+---------------+----------------------------------+-------------
+-----
Vol2 | async_created | 855C9BA51991B0CC122A3791996F6B15 | 512 |
...

For scripting, pass the -t SeparatorCharacters option. This will make it easier to parse the
output.

221
St orage Administ rat ion G uide

Examp le 26 .4 . Scrip t in g examp les

$ lsmcli list --type volumes -t#

Vol1#volume_name#049167B5D09EC0A173E92A63F6C3EA2A#512#41943040#21474836
480#OK#sim-01#POO1
Vol2#async_created#3E771A2E807F68A32FA5E15C235B60CC#512#41943040#214748
36480#OK#sim-01#POO1

$ lsmcli list --type volumes -t " | "

Vol1 | volume_name | 049167B5D09EC0A173E92A63F6C3EA2A | 512 | 41943040
| 21474836480 | OK | 21474836480 | sim-01 | POO1
Vol2 | async_created | 3E771A2E807F68A32FA5E15C235B60CC | 512 |
41943040 | 21474836480 | OK | sim-01 | POO1

$ lsmcli list --type volumes -s

---------------------------------------------
ID | Vol1
Name | volume_name
VPD83 | 049167B5D09EC0A173E92A63F6C3EA2A
Block Size | 512
#blocks | 41943040
Size | 21474836480
Status | OK
System ID | sim-01
Pool ID | POO1
---------------------------------------------
ID | Vol2
Name | async_created
VPD83 | 3E771A2E807F68A32FA5E15C235B60CC
Block Size | 512
#blocks | 41943040
Size | 21474836480
Status | OK
System ID | sim-01
Pool ID | POO1
---------------------------------------------

It is recommended to use the Python library for non-trivial scripting.

For more information on l smcl i , refer to the man pages or the command l smcl i --hel p.

222
 Appendix A. Red Hat Cust omer Port al Labs Relevant t o St orage Administ rat ion

Appendix A. Red Hat Customer Portal Labs Relevant to Storage

Administration
Red Hat Customer Portal Labs are tools designed to help you improve performance, troubleshoot
issues, identify security problems, and optimize configuration. This appendix provides an overview
of Red Hat Customer Portal Labs relevant to storage administration. All Red Hat Customer Portal
Labs are available at https://access.redhat.com/labs/.

SCSI decoder

The SCSI decoder is designed to decode SCSI error messages in the /l o g /* files or log file
snippets, as these error messages can be hard to understand for the user.

Use the SC SI d eco d er to individually diagnose each SCSI error message and get solutions to
resolve problems efficiently.

File Syst em Layout Calculat or

The File System Layout Calculator determines the optimal parameters for creating ext3, ext4, and xfs
file systems, after you provide storage options that describe your current or planned storage. Move
the cursor over the question mark (" ?" ) for a brief explanation of a particular option, or scroll down to
read a summary of all options.

Use the File Syst em Layo u t C alcu lat o r to generate a command that creates a file system with
provided parameters on the specified RAID storage. Copy the generated command and execute it as
ro o t to create the required file system.

LVM RAID Calculat or

The LVM RAID Calculator determines the optimal parameters for creating logical volumes (LVMs) on a
given RAID storage after you specify storage options. Move the cursor over the question mark (" ?" )
for a brief explanation of a particular option, or scroll down to read a summary of all options.

The LVM R AID C alcu lat o r generates a sequence of commands that create LVMs on a given RAID
storage. Copy and execute the generated commands one by one as ro o t to create the required
LVMs.

iSCSI Helper

The iSCSI Helper provides a block-level storage over Internet Protocol (IP) networks, and enables the
use of storage pools within server virtualization.

Use the iSC SI H elp er to generate a script that prepares the system for its role of an iSCSI target
(server) or an iSCSI initiator (client) configured according to the settings that you provide.

Samba Configurat ion Helper

The Samba Configuration Helper creates a configuration that provides basic file and printer sharing
through Samba:

Click Server to specify basic server settings.

223
St orage Administ rat ion G uide

Click Shares to add the directories that you want to share

Click Server to add attached printers individually.

Mult ipat h Helper

The Multipath Helper creates an optimal configuration for multipath devices on Red Hat
Enterprise Linux 5, 6, and 7. By following the steps, you can create advanced multipath
configurations, such as custom aliases or device blacklists.

The Multipath Helper also provides the mul ti path. co nf file for a review. When you achieve the
required configuration, download the installation script to run on your server.

NFS Helper

The NFS Helper simplifies configuring a new NFS server or client. Follow the steps to specify the
export and mount options. Then, generate a downloadable NFS configuration script.

Mult ipat h Configurat ion Visualiz er

The Multipath Configuration Visualizer analyzes files in a sosreport and provides a diagram that
visualizes the multipath configuration. Use the Mu lt ip at h C o n f ig u rat io n Visu aliz er to display:

Hosts components including Host Bus Adapters (HBAs), local devices, and iSCSI devices on the
server side

Storage components on the storage side

Fabric or Ethernet components between the server and the storage

Paths to all mentioned components

You can either upload a sosreport compressed in the .xz, .gz, or .bz2 format, or extract a sosreport in
a directory that you then select as the source for a client-side analysis.

RHEL Backup and Rest ore Assist ant

The RHEL Backup and Restore Assistant provides information on back-up and restore tools, and
common scenarios of Linux usage.

D escribed tools:

d ump and resto re: for backing up the ext2, ext3, and ext4 file systems.

tar and cpi o : for archiving or restoring files and folders, especially when backing up the tape
drives.

rsync: for performing back-up operations and synchronizing files and directories between
locations.

d d : for copying files from a source to a destination block by block independently of the file
systems or operating systems involved.

D escribed scenarios:

224
 Appendix A. Red Hat Cust omer Port al Labs Relevant t o St orage Administ rat ion

D isaster recovery

Hardware migration

Partition table backup

Important folder backup

Incremental backup

D ifferential backup

225
St orage Administ rat ion G uide

Appendix B. Revision History

R evisio n 3- 84 T u e Sep 5 2017 Marek Su ch an ek
An asynchronous update.

R evisio n 3- 80 T h u Ju l 27 2017 Milan N avrat il

D ocument version for 7.4 GA publication.

R evisio n 3- 77 Wed May 24 2017 Milan N avrat il

An asynchronous update.

R evisio n 3- 6 8 Fri O ct 21 2016 Milan N avrat il

Version for 7.3 GA publication.

R evisio n 3- 6 7 Fri Ju n 17 2016 Milan N avrat il

An asynchronous update.

R evisio n 3- 6 4 Wed N o v 11 2015 Jan a H eves

Version for 7.2 GA release.

R evisio n 3- 33 Wed Feb 18 2015 Jacq u elyn n East

Version for 7.1 GA

R evisio n 3- 26 Wed Jan 21 2015 Jacq u elyn n East

Added overview of Ceph

R evisio n 3- 22 T h u D ec 4 2014 Jacq u elyn n East

7.1 Beta

R evisio n 3- 4 T h u Ju l 17 2014 Jacq u elyn n East

Added new chapter on targetcli

R evisio n 3- 1 T u e Ju n 3 2014 Jacq u elyn n East

Version for 7.0 GA release

Index

Symbols
/b o o t / d irect o ry, T h e /b o o t / D irect o ry
/d ev/sh m, G at h erin g File Syst em In f o rmat io n
/et c/f st ab , C o n vert in g t o an Ext 3 File Syst em, Mo u n t in g N FS File Syst ems u sin g
/et c/f st ab , Mo u n t in g a File Syst em
/et c/f st ab f ile
- enabling disk quotas with, Enabling Quotas

/lo cal/d irect o ry ( clien t co n f ig u rat io n , mo u n t in g )

- NFS, NFS Client Configuration

226
 Appendix B. Revision Hist ory

/p ro c
- /proc/devices, The /proc Virtual File System
- /proc/filesystems, The /proc Virtual File System
- /proc/mdstat, The /proc Virtual File System
- /proc/mounts, The /proc Virtual File System
- /proc/mounts/, The /proc Virtual File System
- /proc/partitions, The /proc Virtual File System

/p ro c/d evices
- virtual file system (/proc), The /proc Virtual File System

/p ro c/f ilesyst ems

- virtual file system (/proc), The /proc Virtual File System

/p ro c/md st at
- virtual file system (/proc), The /proc Virtual File System

/p ro c/mo u n t s
- virtual file system (/proc), The /proc Virtual File System

/p ro c/mo u n t s/
- virtual file system (/proc), The /proc Virtual File System

/p ro c/p art it io n s
- virtual file system (/proc), The /proc Virtual File System

/remo t e/exp o rt ( clien t co n f ig u rat io n , mo u n t in g )

- NFS, NFS Client Configuration

A
ad d in g p at h s t o a st o rag e d evice, Ad d in g a St o rag e D evice o r Pat h
ad d in g /remo vin g
- LUN (logical unit number), Adding/Removing a Logical Unit Through rescan-scsi-
bus.sh

ad van ced R AID d evice creat io n

- RAID , Advanced RAID D evice Creation

allo cat io n f eat u res

- ext4, The Ext4 File System
- XFS, The XFS File System

An aco n d a su p p o rt
- RAID , RAID Support in the Installer

API, Fib re C h an n el, Fib re C h an n el API

API, iSC SI, iSC SI API
AT A st an d ard s
- I/O alignment and size, ATA

au t o f s , au t o f s, au t o f s C o n f ig u rat io n
- (see also NFS)

227
St orage Administ rat ion G uide

au t o f s versio n 5
- NFS, Improvements in autofs Version 5 over Version 4

B
b acku p /rest o rat io n
- XFS, Backup and Restoration of XFS File Systems

b at t ery- b acked writ e cach es

- write barriers, Battery-Backed Write Caches

b cu ll ( cach e cu ll limit s set t in g s)

- FS-Cache, Setting Cache Cull Limits

b in d in g /u n b in d in g an if ace t o a p o rt al
- offload and interface binding
- iSCSI, Binding/Unbinding an iface to a Portal

b lo ck d evice io ct ls ( u sersp ace access)

- I/O alignment and size, Block D evice ioctls

b lo cked d evice, verif yin g

- Fibre Channel
- modifying link loss behavior, Fibre Channel

b ru n ( cach e cu ll limit s set t in g s)

- FS-Cache, Setting Cache Cull Limits

b st o p ( cach e cu ll limit s set t in g s)

- FS-Cache, Setting Cache Cull Limits

B t rf s
- File System, Btrfs (Technology Preview)

C
cach e b ack- en d
- FS-Cache, FS-Cache

cach e cu ll limit s
- FS-Cache, Setting Cache Cull Limits

cach e limit at io n s wit h N FS

- FS-Cache, Cache Limitations With NFS

cach e set u p
- FS-Cache, Setting Up a Cache

cach e sh arin g
- FS-Cache, Cache Sharing

cach ef iles
- FS-Cache, FS-Cache

cach ef ilesd

228
 Appendix B. Revision Hist ory

- FS-Cache, Setting Up a Cache

C C W, ch an n el co mman d wo rd
- storage considerations during installation, D ASD and zFCP D evices on IBM System
Z

ch an g in g d ev_lo ss_t mo
- Fibre Channel
- modifying link loss behavior, Fibre Channel

C h an g in g t h e read /writ e st at e
- Online logical units, Changing the Read/Write State of an Online Logical Unit

ch an n el co mman d wo rd ( C C W)
- storage considerations during installation, D ASD and zFCP D evices on IBM System
Z

co h eren cy d at a
- FS-Cache, FS-Cache

co mman d t imer ( SC SI)

- Linux SCSI layer, Command Timer

co mman d s
- volume_key, volume_key Commands

co n f ig u rat io n
- discovery
- iSCSI, iSCSI D iscovery Configuration

co n f ig u rin g a t f t p service f o r d iskless clien t s

- diskless systems, Configuring a tftp Service for D iskless Clients

co n f ig u rin g an Et h ern et in t erf ace t o u se FC o E

- FCoE, Configuring a Fibre Channel over Ethernet Interface

co n f ig u rin g D H C P f o r d iskless clien t s

- diskless systems, Configuring D HCP for D iskless Clients

co n f ig u rin g R AID set s

- RAID , Configuring RAID Sets

co n t ro llin g SC SI co mman d t imer an d d evice st at u s

- Linux SCSI layer, Controlling the SCSI Command Timer and D evice Status

creat in g
- ext4, Creating an Ext4 File System
- XFS, Creating an XFS File System

cu mu lat ive mo d e ( xf srest o re)

- XFS, Cumulative Mode for xfsrestore

D
D ASD an d z FC P d evices o n IB M Syst em z

229
St orage Administ rat ion G uide

- storage considerations during installation, D ASD and zFCP D evices on IBM System
Z

d eb u g f s ( o t h er ext 4 f ile syst em u t ilit ies)

- ext4, Other Ext4 File System Utilities

d ep lo ymen t
- solid-state disks, Solid-State D isk D eployment Guidelines

d ep lo ymen t g u id elin es
- solid-state disks, Solid-State D isk D eployment Guidelines

d et ermin in g remo t e p o rt st at es
- Fibre Channel
- modifying link loss behavior, Fibre Channel

d ev d irect o ry, T h e /d ev/ D irect o ry

d evice st at u s
- Linux SCSI layer, D evice States

d evice- map p er mu lt ip at h in g , D M- Mu lt ip at h
d evices, remo vin g , R emo vin g a St o rag e D evice
d ev_lo ss_t mo
- Fibre Channel
- modifying link loss behavior, Fibre Channel

- Fibre Channel API, Fibre Channel API

d ev_lo ss_t mo , ch an g in g
- Fibre Channel
- modifying link loss behavior, Fibre Channel

d f , G at h erin g File Syst em In f o rmat io n

D H C P, co n f ig u rin g
- diskless systems, Configuring D HCP for D iskless Clients

D IF/D IX- en ab led b lo ck d evices

- storage considerations during installation, Block D evices with D IF/D IX Enabled

d irect map su p p o rt ( au t o f s versio n 5)

- NFS, Improvements in autofs Version 5 over Version 4

d irect o ries
- /boot/, The /boot/ D irectory
- /dev/, The /dev/ D irectory
- /etc/, The /etc/ D irectory
- /mnt/, The /mnt/ D irectory
- /opt/, The /opt/ D irectory
- /proc/, The /proc/ D irectory
- /srv/, The /srv/ D irectory
- /sys/, The /sys/ D irectory
- /usr/, The /usr/ D irectory
- /var/, The /var/ D irectory

230
 Appendix B. Revision Hist ory

d irt y lo g s ( rep airin g XFS f ile syst ems)

- XFS, Repairing an XFS File System

d isab lin g N O P- O u t s
- iSCSI configuration, iSCSI Root

d isab lin g writ e cach es

- write barriers, D isabling Write Caches

d isco very
- iSCSI, iSCSI D iscovery Configuration

d isk q u o t as, D isk Q u o t as

- additional resources, D isk Quota References
- assigning per file system, Setting the Grace Period for Soft Limits
- assigning per group, Assigning Quotas per Group
- assigning per user, Assigning Quotas per User
- disabling, Enabling and D isabling
- enabling, Configuring D isk Quotas, Enabling and D isabling
- /etc/fstab, modifying, Enabling Quotas
- creating quota files, Creating the Quota D atabase Files
- quotacheck, running, Creating the Quota D atabase Files

- grace period, Assigning Quotas per User

- hard limit, Assigning Quotas per User
- management of, Managing D isk Quotas
- quotacheck command, using to check, Keeping Quotas Accurate
- reporting, Reporting on D isk Quotas

- soft limit, Assigning Quotas per User

d isk st o rag e ( see d isk q u o t as)

- parted (see parted)

d iskless syst ems

- D HCP, configuring, Configuring D HCP for D iskless Clients
- exported file systems, Configuring an Exported File System for D iskless Clients
- network booting service, Setting Up A Remote D iskless System
- remote diskless systems, Setting Up A Remote D iskless System
- required packages, Setting Up A Remote D iskless System
- tftp service, configuring, Configuring a tftp Service for D iskless Clients

d m- mu lt ip at h
- iSCSI configuration, iSCSI Settings With dm-multipath

d mraid
- RAID , dmraid

d mraid ( co n f ig u rin g R AID set s)

- RAID , dmraid

d rivers ( n at ive) , Fib re C h an n el, N at ive Fib re C h an n el D rivers an d C ap ab ilit ies

d u , G at h erin g File Syst em In f o rmat io n
d u mp levels

231
St orage Administ rat ion G uide

- XFS, Backup and Restoration of XFS File Systems

E
e2f sck, R evert in g t o an Ext 2 File Syst em
e2imag e ( o t h er ext 4 f ile syst em u t ilit ies)
- ext4, Other Ext4 File System Utilities

e2lab el
- ext4, Other Ext4 File System Utilities

e2lab el ( o t h er ext 4 f ile syst em u t ilit ies)

- ext4, Other Ext4 File System Utilities

en ab lin d /d isab lin g

- write barriers, Enabling/D isabling Write Barriers

en h an ced LD AP su p p o rt ( au t o f s versio n 5)
- NFS, Improvements in autofs Version 5 over Version 4

erro r messag es
- write barriers, Enabling/D isabling Write Barriers

et c d irect o ry, T h e /et c/ D irect o ry

exp ert mo d e ( xf s_q u o t a)
- XFS, XFS Quota Management

exp o rt ed f ile syst ems

- diskless systems, Configuring an Exported File System for D iskless Clients

ext 2
- reverting from ext3, Reverting to an Ext2 File System

ext 3
- converting from ext2, Converting to an Ext3 File System
- creating, Creating an Ext3 File System
- features, The Ext3 File System

ext 4
- allocation features, The Ext4 File System
- creating, Creating an Ext4 File System
- debugfs (other ext4 file system utilities), Other Ext4 File System Utilities
- e2image (other ext4 file system utilities), Other Ext4 File System Utilities
- e2label, Other Ext4 File System Utilities
- e2label (other ext4 file system utilities), Other Ext4 File System Utilities
- file system types, The Ext4 File System
- fsync(), The Ext4 File System
- main features, The Ext4 File System
- mkfs.ext4, Creating an Ext4 File System
- mounting, Mounting an Ext4 File System
- nobarrier mount option, Mounting an Ext4 File System
- other file system utilities, Other Ext4 File System Utilities
- quota (other ext4 file system utilities), Other Ext4 File System Utilities
- resize2fs (resizing ext4), Resizing an Ext4 File System
- resizing, Resizing an Ext4 File System

232
 Appendix B. Revision Hist ory

- stride (specifying stripe geometry), Creating an Ext4 File System

- stripe geometry, Creating an Ext4 File System
- stripe-width (specifying stripe geometry), Creating an Ext4 File System
- tune2fs (mounting), Mounting an Ext4 File System
- write barriers, Mounting an Ext4 File System

F
f ast _io _f ail_t mo
- Fibre Channel API, Fibre Channel API

FC o E
- configuring an Ethernet interface to use FCoE, Configuring a Fibre Channel over
Ethernet Interface
- Fibre Channel over Ethernet, Configuring a Fibre Channel over Ethernet Interface
- required packages, Configuring a Fibre Channel over Ethernet Interface

FH S, O verview o f Filesyst em H ierarch y St an d ard ( FH S) , FH S O rg an iz at io n

- (see also file system)

Fib re C h an n el
- online storage, Fibre Channel

Fib re C h an n el API, Fib re C h an n el API

Fib re C h an n el d rivers ( n at ive) , N at ive Fib re C h an n el D rivers an d C ap ab ilit ies
Fib re C h an n el o ver Et h ern et
- FCoE, Configuring a Fibre Channel over Ethernet Interface

f ile syst em
- FHS standard, FHS Organization
- hierarchy, Overview of Filesystem Hierarchy Standard (FHS)
- organization, FHS Organization
- structure, File System Structure and Maintenance

File Syst em
- Btrfs, Btrfs (Technology Preview)

f ile syst em t yp es
- ext4, The Ext4 File System
- GFS2, Global File System 2
- XFS, The XFS File System

f ile syst ems, G at h erin g File Syst em In f o rmat io n

- ext2 (see ext2)
- ext3 (see ext3)

f in d mn t ( co mman d )
- listing mounts, Listing Currently Mounted File Systems

FS- C ach e
- bcull (cache cull limits settings), Setting Cache Cull Limits
- brun (cache cull limits settings), Setting Cache Cull Limits
- bstop (cache cull limits settings), Setting Cache Cull Limits
- cache back-end, FS-Cache
- cache cull limits, Setting Cache Cull Limits

233
St orage Administ rat ion G uide

- cache sharing, Cache Sharing

- cachefiles, FS-Cache
- cachefilesd, Setting Up a Cache
- coherency data, FS-Cache
- indexing keys, FS-Cache
- NFS (cache limitations with), Cache Limitations With NFS
- NFS (using with), Using the Cache With NFS
- performance guarantee, Performance Guarantee
- setting up a cache, Setting Up a Cache
- statistical information (tracking), Statistical Information
- tune2fs (setting up a cache), Setting Up a Cache

f syn c( )
- ext4, The Ext4 File System
- XFS, The XFS File System

G
G FS2
- file system types, Global File System 2
- gfs2.ko, Global File System 2
- maximum size, Global File System 2

G FS2 f ile syst em maximu m siz e, G lo b al File Syst em 2

g f s2.ko
- GFS2, Global File System 2

G lo b al File Syst em 2
- file system types, Global File System 2
- gfs2.ko, Global File System 2
- maximum size, Global File System 2

g q u o t a/g q n o en f o rce
- XFS, XFS Quota Management

H
H ard ware R AID ( see R AID )
h ard ware R AID co n t ro ller d rivers
- RAID , Linux Hardware RAID controller drivers

h ierarch y, f ile syst em, O verview o f Filesyst em H ierarch y St an d ard ( FH S)

h ig h - en d arrays
- write barriers, High-End Arrays

h o st
- Fibre Channel API, Fibre Channel API

h o w writ e b arriers wo rk
- write barriers, How Write Barriers Work

I
I/O alig n men t an d siz e, St o rag e I/O Alig n men t an d Siz e
- ATA standards, ATA
- block device ioctls (userspace access), Block D evice ioctls

234
 Appendix B. Revision Hist ory

- Linux I/O stack, Storage I/O Alignment and Size

- logical_block_size, Userspace Access
- LVM, Logical Volume Manager
- READ CAPACITY(16), SCSI
- SCSI standards, SCSI
- stacking I/O parameters, Stacking I/O Parameters
- storage access parameters, Parameters for Storage Access
- sysfs interface (userspace access), sysfs Interface
- tools (for partitioning and other file system functions), Partition and File System
Tools
- userspace access, Userspace Access

I/O p aramet ers st ackin g

- I/O alignment and size, Stacking I/O Parameters

if ace ( co n f ig u rin g f o r iSC SI o f f lo ad )

- offload and interface binding
- iSCSI, Configuring an iface for iSCSI Offload

if ace b in d in g /u n b in d in g
- offload and interface binding
- iSCSI, Binding/Unbinding an iface to a Portal

if ace co n f ig u rat io n s, viewin g

- offload and interface binding
- iSCSI, Viewing Available iface Configurations

if ace f o r so f t ware iSC SI

- offload and interface binding
- iSCSI, Configuring an iface for Software iSCSI

if ace set t in g s
- offload and interface binding
- iSCSI, Viewing Available iface Configurations

imp o rt an ce o f writ e b arriers

- write barriers, Importance of Write Barriers

in creasin g f ile syst em siz e

- XFS, Increasing the Size of an XFS File System

in d exin g keys
- FS-Cache, FS-Cache

in d ivid u al u ser
- volume_key, Using volume_key as an individual user

in it iat o r imp lemen t at io n s

- offload and interface binding
- iSCSI, Viewing Available iface Configurations

in st allat io n st o rag e co n f ig u rat io n s

- channel command word (CCW), D ASD and zFCP D evices on IBM System Z
- D ASD and zFCP devices on IBM System z, D ASD and zFCP D evices on IBM System
Z

235
St orage Administ rat ion G uide

- D IF/D IX-enabled block devices, Block D evices with D IF/D IX Enabled

- iSCSI detection and configuration, iSCSI D etection and Configuration
- LUKS/dm-crypt, encrypting block devices using, Encrypting Block D evices Using
LUKS
- separate partitions (for /home, /opt, /usr/local), Separate Partitions for /home, /opt,
/usr/local
- stale BIOS RAID metadata, Stale BIOS RAID Metadata
- updates, Storage Considerations D uring Installation
- what's new, Storage Considerations D uring Installation

in st aller su p p o rt
- RAID , RAID Support in the Installer

in t eract ive o p erat io n ( xf srest o re)

- XFS, Interactive Operation

in t erco n n ect s ( scan n in g )

- iSCSI, Scanning iSCSI Interconnects

in t ro d u ct io n , O verview
iSC SI
- discovery, iSCSI D iscovery Configuration
- configuration, iSCSI D iscovery Configuration
- record types, iSCSI D iscovery Configuration

- offload and interface binding, Configuring iSCSI Offload and Interface Binding
- binding/unbinding an iface to a portal, Binding/Unbinding an iface to a
Portal
- iface (configuring for iSCSI offload), Configuring an iface for iSCSI Offload
- iface configurations, viewing, Viewing Available iface Configurations
- iface for software iSCSI, Configuring an iface for Software iSCSI
- iface settings, Viewing Available iface Configurations
- initiator implementations, Viewing Available iface Configurations
- software iSCSI, Configuring an iface for Software iSCSI
- viewing available iface configurations, Viewing Available iface
Configurations

- scanning interconnects, Scanning iSCSI Interconnects

- software iSCSI, Configuring an iface for Software iSCSI
- targets, Logging in to an iSCSI Target
- logging in, Logging in to an iSCSI Target

iSC SI API, iSC SI API

iSC SI d et ect io n an d co n f ig u rat io n
- storage considerations during installation, iSCSI D etection and Configuration

iSC SI lo g ical u n it , resiz in g , R esiz in g an iSC SI Lo g ical U n it

iSC SI ro o t
- iSCSI configuration, iSCSI Root

issu e_lip
- Fibre Channel API, Fibre Channel API

236
 Appendix B. Revision Hist ory

kn o wn issu es
- adding/removing
- LUN (logical unit number), Known Issues With rescan-scsi-bus.sh

L
laz y mo u n t /u n mo u n t su p p o rt ( au t o f s versio n 5)
- NFS, Improvements in autofs Version 5 over Version 4

levels
- RAID , RAID Levels and Linear Support

limit ( xf s_q u o t a exp ert mo d e)

- XFS, XFS Quota Management

lin ear R AID

- RAID , RAID Levels and Linear Support

Lin u x I/O st ack

- I/O alignment and size, Storage I/O Alignment and Size

lo g g in g in
- iSCSI targets, Logging in to an iSCSI Target

lo g ical_b lo ck_siz e
- I/O alignment and size, Userspace Access

LU K S/d m- cryp t , en cryp t in g b lo ck d evices u sin g

- storage considerations during installation, Encrypting Block D evices Using LUKS

LU N ( lo g ical u n it n u mb er)
- adding/removing, Adding/Removing a Logical Unit Through rescan-scsi-bus.sh
- known issues, Known Issues With rescan-scsi-bus.sh
- required packages, Adding/Removing a Logical Unit Through rescan-scsi-
bus.sh
- rescan-scsi-bus.sh, Adding/Removing a Logical Unit Through rescan-scsi-
bus.sh

LVM
- I/O alignment and size, Logical Volume Manager

M
main f eat u res
- ext4, The Ext4 File System
- XFS, The XFS File System

maximu m siz e
- GFS2, Global File System 2

maximu m siz e, G FS2 f ile syst em, G lo b al File Syst em 2

md ad m ( co n f ig u rin g R AID set s)
- RAID , mdadm

md raid

237
St orage Administ rat ion G uide

- RAID , mdraid

mirro rin g
- RAID , RAID Levels and Linear Support

mkf s , Fo rmat t in g an d Lab elin g t h e Part it io n

mkf s.ext 4
- ext4, Creating an Ext4 File System

mkf s.xf s
- XFS, Creating an XFS File System

mkp art , Makin g t h e Part it io n

mn t d irect o ry, T h e /mn t / D irect o ry
mo d if yin g lin k lo ss b eh avio r, Mo d if yin g Lin k Lo ss B eh avio r
- Fibre Channel, Fibre Channel

mo u n t ( clien t co n f ig u rat io n )
- NFS, NFS Client Configuration

mo u n t ( co mman d ) , U sin g t h e mo u n t C o mman d

- listing mounts, Listing Currently Mounted File Systems
- mounting a file system, Mounting a File System
- moving a mount point, Moving a Mount Point
- options, Specifying the Mount Options
- shared subtrees, Sharing Mounts
- private mount, Sharing Mounts
- shared mount, Sharing Mounts
- slave mount, Sharing Mounts
- unbindable mount, Sharing Mounts

mo u n t in g , Mo u n t in g a File Syst em
- ext4, Mounting an Ext4 File System
- XFS, Mounting an XFS File System

mo vin g a mo u n t p o in t , Mo vin g a Mo u n t Po in t
mu lt ip le mast er map en t ries p er au t o f s mo u n t p o in t ( au t o f s versio n 5)
- NFS, Improvements in autofs Version 5 over Version 4

N
n at ive Fib re C h an n el d rivers, N at ive Fib re C h an n el D rivers an d C ap ab ilit ies
n et wo rk b o o t in g service
- diskless systems, Setting Up A Remote D iskless System

N et wo rk File Syst em ( see N FS)

N FS
- /etc/fstab , Mounting NFS File Systems using /etc/fstab
- /local/directory (client configuration, mounting), NFS Client Configuration
- /remote/export (client configuration, mounting), NFS Client Configuration
- additional resources, References
- installed documentation, Installed D ocumentation
- related books, Related Books
- useful websites, Useful Websites

238
 Appendix B. Revision Hist ory

- autofs
- augmenting, Overriding or Augmenting Site Configuration Files
- configuration, autofs Configuration
- LD AP, Using LD AP to Store Automounter Maps

- autofs version 5, Improvements in autofs Version 5 over Version 4

- client
- autofs , autofs
- configuration, NFS Client Configuration
- mount options, Common NFS Mount Options

- condrestart, Starting and Stopping NFS

- configuration with firewall, Running NFS Behind a Firewall
- direct map support (autofs version 5), Improvements in autofs Version 5 over Version
4
- enhanced LD AP support (autofs version 5), Improvements in autofs Version 5 over
Version 4
- FS-Cache, Using the Cache With NFS
- hostname formats, Hostname Formats
- how it works, How NFS Works
- introducing, Network File System (NFS)
- lazy mount/unmount support (autofs version 5), Improvements in autofs Version 5
over Version 4
- mount (client configuration), NFS Client Configuration
- multiple master map entries per autofs mount point (autofs version 5), Improvements
in autofs Version 5 over Version 4
- options (client configuration, mounting), NFS Client Configuration
- overriding/augmenting site configuration files (autofs), autofs Configuration
- proper nsswitch configuration (autofs version 5), use of, Improvements in autofs
Version 5 over Version 4
- RD MA, Enabling NFS over RD MA (NFSoRD MA)
- reloading, Starting and Stopping NFS
- required services, Required Services
- restarting, Starting and Stopping NFS
- rfc2307bis (autofs), Using LD AP to Store Automounter Maps
- rpcbind , NFS and rpcbind
- security, Securing NFS
- file permissions, File Permissions
- NFSv3 host access, NFS Security with AUTH_SYS and export controls
- NFSv4 host access, NFS security with AUTH_GSS

- server (client configuration, mounting), NFS Client Configuration

- server configuration, NFS Server Configuration
- /etc/exports , The /etc/exports Configuration File
- exportfs command, The exportfs Command
- exportfs command with NFSv4, Using exportfs with NFSv4

- starting, Starting and Stopping NFS

- status, Starting and Stopping NFS
- stopping, Starting and Stopping NFS
- storing automounter maps, using LD AP to store (autofs), Overriding or Augmenting
Site Configuration Files
- TCP, How NFS Works
- troubleshooting NFS and rpcbind, Troubleshooting NFS and rpcbind
- UD P, How NFS Works
- write barriers, NFS

239
St orage Administ rat ion G uide

N FS ( cach e limit at io n s wit h )

- FS-Cache, Cache Limitations With NFS

N FS ( u sin g wit h )
- FS-Cache, Using the Cache With NFS

n o b arrier mo u n t o p t io n
- ext4, Mounting an Ext4 File System
- XFS, Write Barriers

N O P- O u t req u est s
- modifying link loss
- iSCSI configuration, NOP-Out Interval/Timeout

N O P- O u t s ( d isab lin g )
- iSCSI configuration, iSCSI Root

O
o f f lin e st at u s
- Linux SCSI layer, Controlling the SCSI Command Timer and D evice Status

o f f lo ad an d in t erf ace b in d in g
- iSCSI, Configuring iSCSI Offload and Interface Binding

O n lin e lo g ical u n it s
- Changing the read/write state, Changing the Read/Write State of an Online Logical
Unit

o n lin e st o rag e
- Fibre Channel, Fibre Channel
- overview, Online Storage Management
- sysfs, Online Storage Management

- troubleshooting, Online Storage Configuration Troubleshooting

o p t d irect o ry, T h e /o p t / D irect o ry

o p t io n s ( clien t co n f ig u rat io n , mo u n t in g )
- NFS, NFS Client Configuration

o t h er f ile syst em u t ilit ies

- ext4, Other Ext4 File System Utilities

o verrid in g /au g men t in g sit e co n f ig u rat io n f iles ( au t o f s)

- NFS, autofs Configuration

o verview, O verview
- online storage, Online Storage Management

P
Parallel N FS
- pNFS, pNFS

p aramet ers f o r st o rag e access

24 0
 Appendix B. Revision Hist ory

- I/O alignment and size, Parameters for Storage Access

p arit y
- RAID , RAID Levels and Linear Support

p art ed , Part it io n s
- creating partitions, Creating a Partition
- overview, Partitions
- removing partitions, Removing a Partition
- resizing partitions, Resizing a Partition with fdisk
- selecting device, Viewing the Partition Table
- table of commands, Partitions
- viewing partition table, Viewing the Partition Table

p art it io n t ab le
- viewing, Viewing the Partition Table

p art it io n s
- creating, Creating a Partition
- formatting
- mkfs , Formatting and Labeling the Partition

- making
- mkpart , Making the Partition

- removing, Removing a Partition

- resizing, Resizing a Partition with fdisk
- viewing list, Viewing the Partition Table

p at h t o st o rag e d evices, ad d in g , Ad d in g a St o rag e D evice o r Pat h

p at h t o st o rag e d evices, remo vin g , R emo vin g a Pat h t o a St o rag e D evice
p erf o rman ce g u aran t ee
- FS-Cache, Performance Guarantee

p ersist en t n amin g , Persist en t N amin g

p N FS
- Parallel NFS, pNFS

p o rt st at es ( remo t e) , d et ermin in g
- Fibre Channel
- modifying link loss behavior, Fibre Channel

p q u o t a/p q n o en f o rce
- XFS, XFS Quota Management

p rivat e mo u n t , Sh arin g Mo u n t s
p ro c d irect o ry, T h e /p ro c/ D irect o ry
p ro ject limit s ( set t in g )
- XFS, Setting Project Limits

p ro p er n sswit ch co n f ig u rat io n ( au t o f s versio n 5) , u se o f

- NFS, Improvements in autofs Version 5 over Version 4

24 1
St orage Administ rat ion G uide

q u eu e_if _n o _p at h
- iSCSI configuration, iSCSI Settings With dm-multipath
- modifying link loss
- iSCSI configuration, replacement_timeout

q u o t a ( o t h er ext 4 f ile syst em u t ilit ies)

- ext4, Other Ext4 File System Utilities

q u o t a man ag emen t
- XFS, XFS Quota Management

q u o t ach eck , C reat in g t h e Q u o t a D at ab ase Files

q u o t ach eck co mman d
- checking quota accuracy with, Keeping Quotas Accurate

q u o t ao f f , En ab lin g an d D isab lin g

q u o t ao n , En ab lin g an d D isab lin g

R
R AID
- advanced RAID device creation, Advanced RAID D evice Creation
- Anaconda support, RAID Support in the Installer
- configuring RAID sets, Configuring RAID Sets
- dmraid, dmraid
- dmraid (configuring RAID sets), dmraid
- Hardware RAID , RAID Types
- hardware RAID controller drivers, Linux Hardware RAID controller drivers
- installer support, RAID Support in the Installer
- level 0, RAID Levels and Linear Support
- level 1, RAID Levels and Linear Support
- level 4, RAID Levels and Linear Support
- level 5, RAID Levels and Linear Support
- levels, RAID Levels and Linear Support
- linear RAID , RAID Levels and Linear Support
- mdadm (configuring RAID sets), mdadm
- mdraid, mdraid
- mirroring, RAID Levels and Linear Support
- parity, RAID Levels and Linear Support
- reasons to use, Redundant Array of Independent D isks (RAID )
- Software RAID , RAID Types
- striping, RAID Levels and Linear Support
- subsystems of RAID , Linux RAID Subsystems

R D MA
- NFS, Enabling NFS over RD MA (NFSoRD MA)

R EAD C APAC IT Y( 16 )
- I/O alignment and size, SCSI

reco rd t yp es
- discovery
- iSCSI, iSCSI D iscovery Configuration

R ed H at En t erp rise Lin u x- sp ecif ic f ile lo cat io n s

24 2
 Appendix B. Revision Hist ory

- /etc/sysconfig/, Special Red Hat Enterprise Linux File Locations

- (see also sysconfig directory)

- /var/cache/yum, Special Red Hat Enterprise Linux File Locations

- /var/lib/rpm/, Special Red Hat Enterprise Linux File Locations

remo t e d iskless syst ems

- diskless systems, Setting Up A Remote D iskless System

remo t e p o rt
- Fibre Channel API, Fibre Channel API

remo t e p o rt st at es, d et ermin in g

- Fibre Channel
- modifying link loss behavior, Fibre Channel

remo vin g d evices, R emo vin g a St o rag e D evice

remo vin g p at h s t o a st o rag e d evice, R emo vin g a Pat h t o a St o rag e D evice
rep airin g f ile syst em
- XFS, Repairing an XFS File System

rep airin g XFS f ile syst ems wit h d irt y lo g s

- XFS, Repairing an XFS File System

rep lacemen t _t imeo u t

- modifying link loss
- iSCSI configuration, SCSI Error Handler, replacement_timeout

rep lacemen t _t imeo u t M

- iSCSI configuration, iSCSI Root

rep o rt ( xf s_q u o t a exp ert mo d e)

- XFS, XFS Quota Management

req u ired p ackag es

- adding/removing
- LUN (logical unit number), Adding/Removing a Logical Unit Through
rescan-scsi-bus.sh

- diskless systems, Setting Up A Remote D iskless System

- FCoE, Configuring a Fibre Channel over Ethernet Interface

rescan - scsi- b u s.sh

- adding/removing
- LUN (logical unit number), Adding/Removing a Logical Unit Through
rescan-scsi-bus.sh

resiz e2f s, R evert in g t o an Ext 2 File Syst em

resiz e2f s ( resiz in g ext 4 )
- ext4, Resizing an Ext4 File System

resiz ed lo g ical u n it s, resiz in g , R esiz in g an O n lin e Lo g ical U n it

resiz in g
- ext4, Resizing an Ext4 File System

24 3
St orage Administ rat ion G uide

resiz in g an iSC SI lo g ical u n it , R esiz in g an iSC SI Lo g ical U n it

resiz in g resiz ed lo g ical u n it s, R esiz in g an O n lin e Lo g ical U n it
rest o rin g a b acku p
- XFS, Backup and Restoration of XFS File Systems

rf c2307b is ( au t o f s)
- NFS, Using LD AP to Store Automounter Maps

rp cb in d , N FS an d rp cb in d
- (see also NFS)
- NFS, Troubleshooting NFS and rpcbind
- rpcinfo , Troubleshooting NFS and rpcbind
- status, Starting and Stopping NFS

rp cin f o , T ro u b lesh o o t in g N FS an d rp cb in d
ru n n in g sessio n s, ret rievin g in f o rmat io n ab o u t
- iSCSI API, iSCSI API

ru n n in g st at u s
- Linux SCSI layer, Controlling the SCSI Command Timer and D evice Status

S
scan n in g in t erco n n ect s
- iSCSI, Scanning iSCSI Interconnects

scan n in g st o rag e in t erco n n ect s, Scan n in g St o rag e In t erco n n ect s

SC SI co mman d t imer
- Linux SCSI layer, Command Timer

SC SI Erro r H an d ler
- modifying link loss
- iSCSI configuration, SCSI Error Handler

SC SI st an d ard s
- I/O alignment and size, SCSI

sep arat e p art it io n s ( f o r /h o me, /o p t , /u sr/lo cal)

- storage considerations during installation, Separate Partitions for /home, /opt,
/usr/local

server ( clien t co n f ig u rat io n , mo u n t in g )

- NFS, NFS Client Configuration

set t in g u p a cach e
- FS-Cache, Setting Up a Cache

sh ared mo u n t , Sh arin g Mo u n t s
sh ared su b t rees, Sh arin g Mo u n t s
- private mount, Sharing Mounts
- shared mount, Sharing Mounts
- slave mount, Sharing Mounts
- unbindable mount, Sharing Mounts

24 4
 Appendix B. Revision Hist ory

simp le mo d e ( xf srest o re)

- XFS, Simple Mode for xfsrestore

slave mo u n t , Sh arin g Mo u n t s
so f t ware iSC SI
- iSCSI, Configuring an iface for Software iSCSI
- offload and interface binding
- iSCSI, Configuring an iface for Software iSCSI

So f t ware R AID ( see R AID )

so lid - st at e d isks
- deployment, Solid-State D isk D eployment Guidelines
- deployment guidelines, Solid-State D isk D eployment Guidelines
- SSD , Solid-State D isk D eployment Guidelines
- throughput classes, Solid-State D isk D eployment Guidelines
- TRIM command, Solid-State D isk D eployment Guidelines

sp ecif ic sessio n t imeo u t s, co n f ig u rin g

- iSCSI configuration, Configuring Timeouts for a Specific Session

srv d irect o ry, T h e /srv/ D irect o ry

SSD
- solid-state disks, Solid-State D isk D eployment Guidelines

SSM
- System Storage Manager, System Storage Manager (SSM)
- Backends, SSM Backends
- Installation, SSM Installation
- list command, Show information about all detected devices
- resize command, Increase a volume's size
- snapshot command, Snapshot

st ackin g I/O p aramet ers

- I/O alignment and size, Stacking I/O Parameters

st ale B IO S R AID met ad at a

- storage considerations during installation, Stale BIOS RAID Metadata

st at ist ical in f o rmat io n ( t rackin g )

- FS-Cache, Statistical Information

st o rag e access p aramet ers

- I/O alignment and size, Parameters for Storage Access

st o rag e co n sid erat io n s d u rin g in st allat io n

- channel command word (CCW), D ASD and zFCP D evices on IBM System Z
- D ASD and zFCP devices on IBM System z, D ASD and zFCP D evices on IBM System
Z
- D IF/D IX-enabled block devices, Block D evices with D IF/D IX Enabled
- iSCSI detection and configuration, iSCSI D etection and Configuration
- LUKS/dm-crypt, encrypting block devices using, Encrypting Block D evices Using
LUKS

24 5
St orage Administ rat ion G uide

- separate partitions (for /home, /opt, /usr/local), Separate Partitions for /home, /opt,
/usr/local
- stale BIOS RAID metadata, Stale BIOS RAID Metadata
- updates, Storage Considerations D uring Installation
- what's new, Storage Considerations D uring Installation

st o rag e in t erco n n ect s, scan n in g , Scan n in g St o rag e In t erco n n ect s

st o rin g au t o mo u n t er map s, u sin g LD AP t o st o re ( au t o f s)
- NFS, Overriding or Augmenting Site Configuration Files

st rid e ( sp ecif yin g st rip e g eo met ry)

- ext4, Creating an Ext4 File System

st rip e g eo met ry
- ext4, Creating an Ext4 File System

st rip e- wid t h ( sp ecif yin g st rip e g eo met ry)

- ext4, Creating an Ext4 File System

st rip in g
- RAID , RAID Levels and Linear Support
- RAID fundamentals, Redundant Array of Independent D isks (RAID )

su ( mkf s.xf s su b - o p t io n s)
- XFS, Creating an XFS File System

su b syst ems o f R AID

- RAID , Linux RAID Subsystems

su sp en d in g
- XFS, Suspending an XFS File System

sw ( mkf s.xf s su b - o p t io n s)
- XFS, Creating an XFS File System

swap sp ace, Swap Sp ace

- creating, Adding Swap Space
- expanding, Adding Swap Space
- file
- creating, Creating a Swap File, Removing a Swap File

- LVM2
- creating, Creating an LVM2 Logical Volume for Swap
- extending, Extending Swap on an LVM2 Logical Volume
- reducing, Reducing Swap on an LVM2 Logical Volume
- removing, Removing an LVM2 Logical Volume for Swap

- moving, Moving Swap Space

- recommended size, Swap Space
- removing, Removing Swap Space

sys d irect o ry, T h e /sys/ D irect o ry

sysco n f ig d irect o ry, Sp ecial R ed H at En t erp rise Lin u x File Lo cat io n s
sysf s

24 6
 Appendix B. Revision Hist ory

- overview
- online storage, Online Storage Management

sysf s in t erf ace ( u sersp ace access)

- I/O alignment and size, sysfs Interface

syst em in f o rmat io n
- file systems, Gathering File System Information
- /dev/shm, Gathering File System Information

Syst em St o rag e Man ag er

- SSM, System Storage Manager (SSM)
- Backends, SSM Backends
- Installation, SSM Installation
- list command, Show information about all detected devices
- resize command, Increase a volume's size
- snapshot command, Snapshot

T
t arg et s
- iSCSI, Logging in to an iSCSI Target

t f t p service, co n f ig u rin g
- diskless systems, Configuring a tftp Service for D iskless Clients

t h ro u g h p u t classes
- solid-state disks, Solid-State D isk D eployment Guidelines

t imeo u t s f o r a sp ecif ic sessio n , co n f ig u rin g

- iSCSI configuration, Configuring Timeouts for a Specific Session

t o o ls ( f o r p art it io n in g an d o t h er f ile syst em f u n ct io n s)

- I/O alignment and size, Partition and File System Tools

t rackin g st at ist ical in f o rmat io n

- FS-Cache, Statistical Information

t ran sp o rt
- Fibre Channel API, Fibre Channel API

T R IM co mman d
- solid-state disks, Solid-State D isk D eployment Guidelines

t ro u b lesh o o t in g
- online storage, Online Storage Configuration Troubleshooting

t ro u b lesh o o t in g N FS an d rp cb in d
- NFS, Troubleshooting NFS and rpcbind

t u n e2f s
- converting to ext3 with, Converting to an Ext3 File System
- reverting to ext2 with, Reverting to an Ext2 File System

t u n e2f s ( mo u n t in g )

24 7
St orage Administ rat ion G uide

- ext4, Mounting an Ext4 File System

t u n e2f s ( set t in g u p a cach e)

- FS-Cache, Setting Up a Cache

U
u d ev ru le ( t imeo u t )
- command timer (SCSI), Command Timer

u mo u n t , U n mo u n t in g a File Syst em
u n b in d ab le mo u n t , Sh arin g Mo u n t s
u n mo u n t in g , U n mo u n t in g a File Syst em
u p d at es
- storage considerations during installation, Storage Considerations D uring
Installation

u q u o t a/u q n o en f o rce
- XFS, XFS Quota Management

u sersp ace access

- I/O alignment and size, Userspace Access

u sersp ace API f iles

- Fibre Channel API, Fibre Channel API

u sr d irect o ry, T h e /u sr/ D irect o ry

V
var d irect o ry, T h e /var/ D irect o ry
var/lib /rp m/ d irect o ry, Sp ecial R ed H at En t erp rise Lin u x File Lo cat io n s
var/sp o o l/u p 2d at e/ d irect o ry, Sp ecial R ed H at En t erp rise Lin u x File Lo cat io n s
verif yin g if a d evice is b lo cked
- Fibre Channel
- modifying link loss behavior, Fibre Channel

versio n
- what is new
- autofs, Improvements in autofs Version 5 over Version 4

viewin g availab le if ace co n f ig u rat io n s

- offload and interface binding
- iSCSI, Viewing Available iface Configurations

virt u al f ile syst em ( /p ro c)

- /proc/devices, The /proc Virtual File System
- /proc/filesystems, The /proc Virtual File System
- /proc/mdstat, The /proc Virtual File System
- /proc/mounts, The /proc Virtual File System
- /proc/mounts/, The /proc Virtual File System
- /proc/partitions, The /proc Virtual File System

virt u al st o rag e, Virt u al St o rag e

24 8
 Appendix B. Revision Hist ory

vo lu me_key
- commands, volume_key Commands
- individual user, Using volume_key as an individual user

W
wh at ' s n ew
- storage considerations during installation, Storage Considerations D uring
Installation

Wo rld Wid e Id en t if ier ( WWID )

- persistent naming, WWID

writ e b arriers
- battery-backed write caches, Battery-Backed Write Caches
- definition, Write Barriers
- disabling write caches, D isabling Write Caches
- enablind/disabling, Enabling/D isabling Write Barriers
- error messages, Enabling/D isabling Write Barriers
- ext4, Mounting an Ext4 File System
- high-end arrays, High-End Arrays
- how write barriers work, How Write Barriers Work
- importance of write barriers, Importance of Write Barriers
- NFS, NFS
- XFS, Write Barriers

writ e cach es, d isab lin g

- write barriers, D isabling Write Caches

WWID
- persistent naming, WWID

X
XFS
- allocation features, The XFS File System
- backup/restoration, Backup and Restoration of XFS File Systems
- creating, Creating an XFS File System
- cumulative mode (xfsrestore), Cumulative Mode for xfsrestore
- dump levels, Backup and Restoration of XFS File Systems
- expert mode (xfs_quota), XFS Quota Management
- file system types, The XFS File System
- fsync(), The XFS File System
- gquota/gqnoenforce, XFS Quota Management
- increasing file system size, Increasing the Size of an XFS File System
- interactive operation (xfsrestore), Interactive Operation
- limit (xfs_quota expert mode), XFS Quota Management
- main features, The XFS File System
- mkfs.xfs, Creating an XFS File System
- mounting, Mounting an XFS File System
- nobarrier mount option, Write Barriers
- pquota/pqnoenforce, XFS Quota Management
- project limits (setting), Setting Project Limits
- quota management, XFS Quota Management
- repairing file system, Repairing an XFS File System
- repairing XFS file systems with dirty logs, Repairing an XFS File System
- report (xfs_quota expert mode), XFS Quota Management

24 9
St orage Administ rat ion G uide

- simple mode (xfsrestore), Simple Mode for xfsrestore

- su (mkfs.xfs sub-options), Creating an XFS File System
- suspending, Suspending an XFS File System
- sw (mkfs.xfs sub-options), Creating an XFS File System
- uquota/uqnoenforce, XFS Quota Management
- write barriers, Write Barriers
- xfsdump, Backup and Restoration of XFS File Systems
- xfsprogs, Suspending an XFS File System
- xfsrestore, Backup and Restoration of XFS File Systems
- xfs_admin, Other XFS File System Utilities
- xfs_bmap, Other XFS File System Utilities
- xfs_copy, Other XFS File System Utilities
- xfs_db, Other XFS File System Utilities
- xfs_freeze, Suspending an XFS File System
- xfs_fsr, Other XFS File System Utilities
- xfs_growfs, Increasing the Size of an XFS File System
- xfs_info, Other XFS File System Utilities
- xfs_mdrestore, Other XFS File System Utilities
- xfs_metadump, Other XFS File System Utilities
- xfs_quota, XFS Quota Management
- xfs_repair, Repairing an XFS File System

xf sd u mp
- XFS, Backup and Restoration of XFS File Systems

xf sp ro g s
- XFS, Suspending an XFS File System

xf srest o re
- XFS, Backup and Restoration of XFS File Systems

xf s_ad min
- XFS, Other XFS File System Utilities

xf s_b map
- XFS, Other XFS File System Utilities

xf s_co p y
- XFS, Other XFS File System Utilities

xf s_d b
- XFS, Other XFS File System Utilities

xf s_f reez e
- XFS, Suspending an XFS File System

xf s_f sr
- XFS, Other XFS File System Utilities

xf s_g ro wf s
- XFS, Increasing the Size of an XFS File System

xf s_in f o
- XFS, Other XFS File System Utilities

250
 Appendix B. Revision Hist ory

xf s_md rest o re
- XFS, Other XFS File System Utilities

xf s_met ad u mp
- XFS, Other XFS File System Utilities

xf s_q u o t a
- XFS, XFS Quota Management

xf s_rep air
- XFS, Repairing an XFS File System

251