Scribd
Upload
137 views

Corso GPFS Part 2

This document provides a tutorial on GPFS v.3.4. It discusses intelligent life-cycle management (ILM) including storage pools, filesets, policies and performance tuning. Storage pools allow creation of disk groups and management of data placement. Filesets define namespaces and administrative boundaries. Policies automate data placement and migration between storage tiers. Performance tuning topics include block size selection, metadata/data separation, and TCP/IP settings for 10GbE networks.

Uploaded by

sagar.srivastava
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
137 views

Corso GPFS Part 2

This document provides a tutorial on GPFS v.3.4. It discusses intelligent life-cycle management (ILM) including storage pools, filesets, policies and performance tuning. Storage pools allow creation of disk groups and management of data placement. Filesets define namespaces and administrative boundaries. Policies automate data placement and migration between storage tiers. Performance tuning topics include block size selection, metadata/data separation, and TCP/IP settings for 10GbE networks.

Uploaded by

sagar.srivastava
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Tutorial for GPFS v.3.

12/3/12

V.Sapunenko - GPFS tutorial

Outline
Part 2:

Intelligent Life-circle Management (ILM)


Storage pools and fileset
Policies
Performance tuning

12/4/12

V.Sapunenko - GPFS tutorial

Information Lifecycle
Management (ILM)

policy-driven automation and tiered storage management


storage pools

filesets

Storage pools allow the creation of disk groups within a file


system (hardware partitioning)
new feature in v. 3.2 - external storage pools (to interact with an
external storage management, e.g. TSM)
sub-tree of the file system namespace
administrative boundary to set quotas
control initial data placement or data migration

user-defined policies

12/4/12

V.Sapunenko - GPFS tutorial

Information Lifecycle
Management in GPFS

File placement policies

Define where the data will be created


(appropriate storage pool)
Rules are determined by attributes
like

File name
User name
Fileset

File management policies

Possibility to move data from one pool


to another without changing file
location in the directory structure
Change replication status
Prune file system (deleting files as
defined by policy)
Determined by attributes like

Access time
Path name
Size of the file

12/4/12

V.Sapunenko - GPFS tutorial

Storage Pools

ILM Manages sets of storage called "storage pools


What is a storage pool?

A named subset of disks and tapes


Each file is assigned to a storage pool based upon policy rules

placement policies (where to place files upon creation)


migration policies (moving files from one pool to another)
deletion policies (removing files from the storage system)

What are they are good for?

Tiered storage (files aged to slower/cheaper disk)


Dedicated storage (e.g., per user or per project or per directory subtree)
Failure containment

To limit the amount of data lost due to a failure


To bound the performance impact of RAID rebuild

Appropriate use of special-purpose storage

Different RAID levels Enterprise grade disk vs. consumer-grade disk


Multimedia friendly storage

12/4/12

V.Sapunenko - GPFS tutorial

GPFS Filesets

What they are:

A named subtree of a GPFS file system


Somewhat like a distinct file system, i.e. a fileset can be unlinked
without

deleting it, and it can subsequently be linked using its name as a handle

What they are good for:

Filesets can have quotas associated with them (global; not per-pool).

- Fileset quotas are independent of user and group quotas

Filesets can be used to restrict the effect of policies to specific files

Side effects:

Unlinked filesets can confuse programs that scan the file system
(e.g.incremental backup programs)
Moving and linking between filesets is not allowed, in keeping with their
being like little file systems

12/4/12

V.Sapunenko - GPFS tutorial

GPFS ILM/HSM Integration

GPFS Integrates its ILM Policies with tape based HSM


Products

GPFS extends its Information Lifecycle Management (ILM)


functionality to integrate with HSM (Hierarchical Storage
System) products.

Supported HSM products include

A single set of policies is used to move data between GPFS storage


pools and tape storage pools.
High Performance Storage System (HPSS)
Tivoli Storage Manager (TSM)

Cool Feature: very fast file scans

1 million files in 13 seconds


1 billion files in 75 minutes

12/4/12

V.Sapunenko - GPFS tutorial

Policies

Initial placement policy:

Default rule:

SET POOL system

Must be defined

For a file system with SP system dedicated for


metadataOnly

With default rule cant write a single byte

Can be changed at any time using


mmchpolicy fs name!

To list policy use


mmlspolicy fs name -L

12/4/12

!!

V.Sapunenko - GPFS tutorial

Sample GPFS ILM Policy


Statements

12/4/12

V.Sapunenko - GPFS tutorial

More complex example


Example of balanced placement policy for a file system with 4 storage pools:
RULE
RULE
RULE
RULE

'DATA3'
'DATA4'
'DATA5'
'DATA6'

SET
SET
SET
SET

POOL
POOL
POOL
POOL

'data1'
'data2'
'data3'
'data4'

LIMIT(99) WHERE INTEGER(RAND()*40)<10!


LIMIT(99) WHERE INTEGER(RAND()*30)<10!
LIMIT(99) WHERE INTEGER(RAND()*20)<10!
LIMIT(99)!

RULE 'DATA32' SET POOL 'data1' LIMIT(99) WHERE INTEGER(RAND()*30)<10!


RULE 'DATA42' SET POOL 'data2' LIMIT(99) WHERE INTEGER(RAND()*20)<10!
RULE 'DATA52' SET POOL 'data3' LIMIT(99)!
RULE 'DATA33' SET POOL 'data1' LIMIT(99) WHERE INTEGER(RAND()*20)<10!
RULE 'DATA43' SET POOL 'data2' LIMIT(99)!
RULE 'DATA34' SET POOL 'data1!

See documentation at !
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp!
12/4/12

V.Sapunenko - GPFS tutorial

10

Migrate and delete policies

If the storage pool named system has an occupancy percentage above 90%
now, bring the occupancy percentage of system down to 70% by migrating
the largest files to storage pool data:
RULE 'mig1' MIGRATE FROM POOL system' THRESHOLD(90,70)
WEIGHT(KB_ALLOCATED) TO POOL data'

Delete files from the storage pool named system that have not been
accessed in the last 30 days, and are named like temporary files or appear
in any directory that is named tmp:
RULE 'del1' DELETE FROM POOL system' WHERE (DAY
S(CURRENT_TIMESTAMP) DAYS(ACCESS_TIME) > 30) AND (lowe
r(NAME) LIKE '%.tmp' OR PATH_NAME LIKE '%/tmp/%')

12/4/12

V.Sapunenko - GPFS tutorial

11

Execution of migrate policy

Manualy via mmapplypolicy:


mmapplypolicy fs name P policy file I yes!

Via callback (user exit)


mmaddcallback!
cannot be run from a Windows node
mmapplypolicy {Device | Directory} [-A IscanBuckets] [-a IscanThreads] [-B
MaxFiles] [-D yyyy-mm-dd [@hh:mm[:ss]]] [-e]!
[-f FileListPrefix] [-g GlobalWorkDirectory] [-I {yes | defer |
test | prepare}] [-i InputFileList] [-L n] [-M!
name=value...]
[-m ThreadLevel] [-N {all | mount | Nod
e[,Node...] | NodeFile | NodeClass}] [-n DirThreadLevel] [-P!
PolicyFile] [-q] [-r FileListPathname...] [-S SnapshotName] [-s
LocalWorkDirectory] [--single-instance] [--sort-buffer-!
size Size]!

12/4/12

V.Sapunenko - GPFS tutorial

12

Performance Tuning

File system

GPFS supports block sizes from 16 KB to 4MB with a default of


256 KB.

IO Type

Application Examples

Blocksize

Large Sequential IO

Scientific Computing, Digital Media

1MB to 4MB

Relational Database

DB2, Oracle

512kb

Small I/O Sequential

General File Service, File based


Analytics,Email, Web Applications

256kb

Special*

Special

16KB-64KB

*Since GPFS 3.3 there are very few workloads that benefit from a file system blocksize of
16KB or 64KB. If you do not have a chance to test your application performance with various
file system blocksize settings you should use the default of 256KB.
12/4/12

V.Sapunenko - GPFS tutorial

13

File system (cont.)

Data and Metadata separation

Use small and fast disks for metadata

If network adapter on NSD servers gets saturated


dedicated separate NSD servers for metadata
disks

place disks of different sizes in different


Storage pools
scatter block allocation map usually performs
better

12/4/12

V.Sapunenko - GPFS tutorial

14

Pagepool

Default pagepool = 64M

generally to small
If FS blocksize=N*256K (default) use
pagepool=N*64M
256M good starting point
On NSD servers:

30% of pagepool >


nsdThreadsPerDisk*#LUNS*maxBlockSize.

On NSD servers running applications (NFS,


XrootD, etc,) use pagepool>4G
12/4/12

V.Sapunenko - GPFS tutorial

15

Some parameters

maxMBpS

Default 150
Recommended value =2*network BW

maxFilesToCache

Default 1000

Too high for worker nodes


Too low for NFS server

As a rule the total of ((maxFilesToCache + maxStatCache) *


nodes) should not exceed (600,000 * (tokenMemLimit/256M) *
(The number of manager nodes - 1))

maxStatCache

Default 4*maxStatCache
12/4/12

V.Sapunenko - GPFS tutorial

16

10 GigaBit Ethernet
Jumbo Frames

Myri10GE Driver: options myri10ge myri10ge_initial_mtu=9000


Configure 10GigE Interface to 9000MTU (ifcfg <ethx> mtu 9000)

TCP/IP Proc settings

net.ipv4.conf.all.arp_filter=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.ipfrag_low_thresh=196608
net.ipv4.ipfrag_high_thresh=262144
net.ipv4.ip_no_pmtu_disc=0
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_syn_backlog=4096
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_rfc1337=1
net.ipv4.tcp_adv_win_scale=2
net.ipv4.tcp_rmem=4096 87380 16777216
12/4/12

net.ipv4.tcp_wmem=4096 87380 16777216


net.ipv4.tcp_mem=196608 262144 393216
net.core.rmem_default=524287
net.core.wmem_default=524287
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_keepalive_intvl=75
net.ipv4.tcp_keepalive_probes=9
net.ipv4.tcp_keepalive_time=7200
net.core.netdev_max_backlog = 250000

V.Sapunenko - GPFS tutorial

17

TCP Memory (10Gbit Ethernet)


net.ipv4.tcp_rmem=4096 87380 16777216!
net.ipv4.tcp_wmem=4096 87380 16777216!
net.ipv4.tcp_mem=1310720 2621440 3145728!
net.core.rmem_default=524287!
net.core.wmem_default=524287!
net.core.rmem_max=16777216!
net.core.wmem_max=16777216!
net.ipv4.tcp_keepalive_intvl=75!
net.ipv4.tcp_keepalive_probes=9!
net.ipv4.tcp_keepalive_time=7200!
net.core.netdev_max_backlog = 250000!

12/4/12

V.Sapunenko - GPFS tutorial

18

GPFS tuning guide

http://www.ibm.com/developerworks/wikis/display/hpccentral/
GPFS+Tuning+Parameters

12/4/12

V.Sapunenko - GPFS tutorial

19

Acknowledgements
Materials used in this presentation, along with presenters own
experience, have been mainly obtained from the following sources:

GPFS Best Practices Programming, Configuration, Environment and


Performance Perspectives by Raymond L. Paden, Deep Computing, IBM,
2010
An Introduction to GPFS Version 3.2
by Scott Fadden, IBM Corporation, 2007
IBM Cluster Information Center Website: http://publib.boulder.ibm.com

12/4/12

V.Sapunenko - GPFS tutorial

20

You might also like