Distributed File Systems

 File Service architecture

 NFS and AFS

 Enhancements in the file systems

Sukkur IBA University , Sukkur DS - Spring 2018 1

Introduction to distributed file systems
 Sharing of stored information is the most important aspect of
distributed resource sharing
 Architecture and implementation of basic distributed file systems
File system is an OS facility providing a convenient programming
interface to disk storage
Add access control and file locking mechanisms
DFS supports sharing of information in the form of files
Performance and reliability should be at least as good as access
to the files stored on the local disks
Types of storage systems and their properties Fig 8.1
All storage systems rely on caching for a balanced performance
Strict consistency model is difficult to achieve
Approximate consistency models are generally used
Persistence of stored data is an issue
 Characteristics of non-distributed file systems
 Requirements for distributed file systems
 Abstract model of a basic distributed file service
 Some recent developments in the design of file services

Sukkur IBA University , Sukkur DS - Spring 2018 2

Fig. 8.1 Storage systems and their properties

Sharing Persis- Distributed Consistency Example

tence cache/replicas maintenance

Main memory 1 RAM

File system 1 UNIX file system
Distributed file system Sun NFS
Web Web server
Distributed shared memory Ivy (DSM, Ch. 18)
Remote objects (RMI/ORB) 1 CORBA
Persistent object store 1 CORBA Persistent
Object Service
Peer-to-peer storage system 2 OceanStore (Ch. 10)

Types of consistency: 1: strict one-copy consistency ; X : No automatic consistency

√: approximate consistency; 2: considerably weaker guarantees.

Sukkur IBA University , Sukkur DS - Spring 2018 3

Characteristics of file systems
 File systems are responsible for organization, storage,
retrieval, naming, sharing and protection of files
They provide programming interfaces
Files contain both data and attributes Fig. 8.3
A typical layered module structure of a non-distributed file system
Fig. 8.2
Distributed file service requires components to deal with client-
server communication and with distributed naming and location
of files in addition to the above
 File system operations
Summary of the main file system operations available to applications
in UNIX systems Fig. 8.4
Some file state information is stored by the file system for each
running program
Access control for files is applied by the file system

Sukkur IBA University , Sukkur DS - Spring 2018 4

Figure 8.3 File attribute record structure

File length
Creation timestamp
Read timestamp
Write timestamp
Attribute timestamp
Reference count
Owner
File type
Access control list

Sukkur IBA University , Sukkur DS - Spring 2018 5

Figure 8.2 File system modules

Direc tory module: relates file names to file IDs

File module: relates file IDs to partic ular files

Ac cess control module: c hecks permis sion for operation requested

File acc es s module: reads or w rites file data or attributes

Bloc k module: acc es ses and alloc ates disk blocks

Device module: dis k I/O and buffering

Sukkur IBA University , Sukkur DS - Spring 2018 6

Figure 8.4 UNIX file system operations

filedes = open(name, mode) Opens an existing file with the given name.
filedes = creat(name, mode) Creates a new file with the given name.
Both operations deliver a file descriptor referencing the open
file. The mode is read, write or both.
status = close(filedes) Closes the open file filedes.
count = read(filedes, buffer, n) Transfers n bytes from the file referenced by filedes to buffer.
count = write(filedes, buffer, n) Transfers n bytes to the file referenced by filedes from buffer.
Both operations deliver the number of bytes actually transferred
and advance the read-write pointer.
pos = lseek(filedes, offset, Moves the read-write pointer to offset (relative or absolute,
whence) depending on whence).
status = unlink(name) Removes the file name from the directory structure. If the file
has no other names, it is deleted.
status = link(name1, name2) Adds a new name (name2) for a file (name1).
status = stat(name, buffer) Gets the file attributes for file name into buffer.

Sukkur IBA University , Sukkur DS - Spring 2018 7

Distributed file system requirements
 Access and location transparency
 Concurrency control
 Performance
 Scalability
 Fault tolerance
 Security
 Transparency
Access transparency: Client programs should be unaware of
distribution of files
Provide a single set of operations for local and remote files
Location transparency: Client programs should see a uniform file
name space
Mobility transparency: Neither client programs nor system
administration tables need to be changed when files are moved
Performance transparency: Change in the load on the service should
not affect its performance
Scaling transparency: The service should be expandable by
incremental growth

Sukkur IBA University , Sukkur DS - Spring 2018 8

Distributed file system requirements
 Concurrent file updates
Issue of concurrency control: Changes in file by one client should not
interfere with the operation of other clients
simultaneously accessing it
 File replication
Enhances scalability of the service
Enhances fault tolerance
Limited form of replication is supported through caching
 File service interface should cater to heterogeneity
 Fault tolerance
Design can be based on at-most-once invocation semantics to
cope with transient communication failures
It can use at-least-once semantics for idempotent operations
Stateless servers can be restarted and service restored after a
failure without the need to recover the previous state
 Consistency
There is a deviation from one-copy update semantics, due to the
propagation delay when replicated files are updated

Sukkur IBA University , Sukkur DS - Spring 2018 9

Distributed file system requirements
 Security
Access control and authentication may be required for some file
operations
 Efficiency
A distributed file system should provide a service that is
comparable or better than local file systems in performance and
reliability
 Case studies - NFS and AFS
File service architecture – Abstract architectural model
Responsibilities are divided among modules
A Client module: Emulates a conventional file system interface
for application programs
Server modules: Perform operations for clients on directories
and on files

 SUN NFS – 1985

First file service that was designed as a product
The definitions of the key interfaces were placed in the public domain

Sukkur IBA University , Sukkur DS - Spring 2018 10

Distributed file system requirements
 SUN NFS….
Provides transparent access to remote files for client programs
running on UNIX and other systems
Client-server relationship is symmetrical
An important goal of NFS is to achieve high level of support for
hardware and operating system heterogeneity

 Andrew File System - Developed at CMU in 1986

Design reflects an intention to support information sharing on a
large scale by minimizing client-server communication
Transfer entire files between server and client computers
Cache them at client’s site until the server receives a more
up-to-date version
A public domain implementation of AFS is available in the Linux
operating system

Sukkur IBA University , Sukkur DS - Spring 2018 11

File Service architecture
 Three components of the file service structure
A flat file service
A directory service
A client module Fig. 8.5
 Flat file service and directory service provide a comprehensive set of
operations for access to files
 Client module provides a single programming interface with
operations on files - The design is open
 Flat file service is responsible for
Implementing operations on the contents of files
Generating UFIDs (Unique file identifiers) that is used to refer to
files in all requests
 Directory service is responsible for providing a mapping between text
names for files and their UFIDs
 Client module that runs in each client computer, integrates and
extends the operations of a flat file service and the directory service
under a single application programming interface
Holds information about the network location of the flat file server
Implements cache of recently used file blocks at the client

Sukkur IBA University , Sukkur DS - Spring 2018 12

Fig 8.5 Model file service architecture

Lookup
AddName
Client computer UnName Server computer
GetNames

Application Application Directory service

program program

Flat file service

Client module

Read
Write
Create
Delete
GetAttributes
SetAttributes
Sukkur IBA University , Sukkur DS - Spring 2018 13
File Service architecture
 Flat file service interface Fig. 8.6
Most of the operations in the interface of flat file service are repeatable
Suitable for implementation by stateless servers
 Access control
Access rights checks should be performed at the server
Server is vulnerable to forged identities
Result of an access rights checks needs to be retained at the server
To make the server stateless, either of the following may be adopted:
An access check is made whenever a file name is converted to an
UFID and the results are encoded in the form of a capability
A user identity is submitted with every client request and access
checks are performed by the server for every file operation
This a more common approach
None of the above can overcome the forged user identities problem
 Directory service interface Fig 8.7
Translate text names to UFIDs

Sukkur IBA University , Sukkur DS - Spring 2018 14

 Figure 8.6 Flat file service operations

Read(FileId, i, n) -> Data If 1 ≤ i ≤ Length(File): Reads a sequence of up to n items

— throwsBadPosition from a file starting at item i and returns it in Data.
Write(FileId, i, Data) If 1 ≤ i ≤ Length(File)+1: Writes a sequence of Data to a
— throwsBadPosition file, starting at item i, extending the file if necessary.
Create() -> FileId Creates a new file of length 0 and delivers a UFID for it.
Delete(FileId) Removes the file from the file store.
GetAttributes(FileId) -> Attr Returns the file attributes for the file.
SetAttributes(FileId, Attr) Sets the file attributes (only those attributes that are not
Shaded in Figure 8.3).

Sukkur IBA University , Sukkur DS - Spring 2018 15

 Figure 8.7 Directory service operations

Lookup(Dir, Name) -> FileId Locates the text name in the directory and returns the
— throwsNotFound relevant UFID. If Name is not in the directory, throws an
exception.
AddName(Dir, Name, FileId) If Name is not in the directory, adds (Name, File) to the
— throwsNameDuplicate directory and updates the file’s attribute record.
If Name is already in the directory: throws an exception.
UnName(Dir, Name) If Name is in the directory: the entry containing Name is
— throwsNotFound removed from the directory.
If Name is not in the directory: throws an exception.
GetNames(Dir, Pattern) -> NameSeq Returns all the text names in the directory that match the
regular expression Pattern.

Sukkur IBA University , Sukkur DS - Spring 2018 16

File Service architecture

 Hierarchic file system

Tree structure of UNIX can be implemented in the client
module using the flat file and the directory services

 File groups
A collection of files located on a given server
A server may hold several file groups and the groups can be
moved between servers
A file cannot change the group to which it belongs
Introduced as a facility for moving collection of files stored on
removable media between computers
UFID includes a file group identifier component
File group identifiers must be unique throughout the distributed
system
Algorithm should ensure global uniqueness

Sukkur IBA University , Sukkur DS - Spring 2018 17

Case Study: Sun Network File System
 Architecture of NFS Fig 8.8
Supports NFS protocol – A set of remote procedure calls that enable
clients to perform operations on a remote file store
Operating-system-independent - networks of UNIX systems
 NFS server module resides in the kernel of each computer that acts as
an NFS server
Client and server communicate through remote procedure calls
Port mapper service enable clients to bind to services
 Virtual file system
NFS provides access transparency through VFS
Other distributed file system can be integrated into NFS by a virtual
file system (VFS) module
 File identifiers in NFS are called file handles
Filesystem identifier i-node number of file i-node generation number
Filesystem identifier: Unique number allocated to each filesystem when it
is created
i-node generation number: Needed because i-node number in UNIX is
reused after a file is removed

Sukkur IBA University , Sukkur DS - Spring 2018 18

 Figure 8.8 NFS architecture

Client computer Server computer

Application Application
program program
UNIX
system calls
UNIX kernel
UNIX kernel Virtual file system Virtual file system
Local Remote
file system

UNIX NFS NFS UNIX

file file
Other

client server
system system
NFS
protocol

Sukkur IBA University , Sukkur DS - Spring 2018 19

 Fig 8.8 NFS architecture
Client computer

Client computer NFS Server computer

Application
program Client

Application Application Application

Kernel program
program program
UNIX
system calls

UNIX kernel Virtual file system Virtual file system

Operations Operations
on local files on
file system

remote files
UNIX NFS UNIX
NFS NFS Client
file file
Other

client server
system system

NFS
protocol
(remote operations)
Sukkur IBA University , Sukkur DS - Spring 2018 20
*
Sun Network File System
 Virtual file system....
File handles are passed from server to client in the results of lookup,
create and mkdir operations
 Virtual file system layer has one VFS structure for each mounted file
system and one v-node per open file
The v-node contains an indicator to show whether the file is local or
remote
 Client integration
NFS client module provides an interface for use by application
programs
It is integrated with the UNIX kernel and emulates the semantics of
standard UNIX file system primitives precisely
User programs can access files via UNIX system calls
A single client module serves all of the user-level processes
Encryption key can be retained in the kernel
NFS client module cooperates with the VFS in each client machine
Maintains caching of different file blocks
Cache consistency problems arise

Sukkur IBA University , Sukkur DS - Spring 2018 21

Sun Network File System
 Access control and authentication
NFS server is stateless
Each request should be checked for authenticity and access control
There is a security loophole in the access-control mechanism
 NFS server interface
The file and directory services are integrated in a single service
 The Mounting of sub-trees of remote filesystems by clients is
supported by a separate mount service process that runs at user-level
on each NFS server computer Fig. 8.10
Remote filesystems may be hard-mounted or soft-mounted in a client
computer
Hard-mounted: The process is suspended until the request is completed
In case of server failure, the user processes are suspended until
the server is restarted
Soft-mounted: NFS client modules returns a failure indication after a
small number of retries
 In NFS, pathnames are parsed and their translation is performed in an
iterative manner by the client

Sukkur IBA University , Sukkur DS - Spring 2018 22

 Figure 8.10 Local and remote file systems
accessible on an NFS client
Server 1 Client Server 2
(root) (root) (root)

export . .. v munix usr nfs

Remote Remote
people s tudents x s taff users
mount mount

big jon bob . . . jim ann jane joe

Note: The file system mounted at /usr/students in the client is actually the sub-tree located at /export/people in Server 1;
the file system mounted at /usr/staff in the client is actually the sub-tree located at /nfs/users in Server 2.

Sukkur IBA University , Sukkur DS - Spring 2018 23

Sun Network File System
 The automounter was added to NFS to mount remote directory
dynamically whenever an ‘empty’ mount point is referenced by a client
Maintains a table of mount points with a reference to one or more
NFS servers
It behaves like a Local NFS server at the client machine
 Server caching
Caching is used in both client and server computers to achieve better
performance - Implementation in Unix
Read ahead and delayed write optimizes performance further
Sync operation flushes altered pages to disk every 30 seconds
 NFS uses cache at the sever, hence no consistency problem
In order to ensure write persistence in case of server crash, one of the
following options can be used:
Write-through caching: Reply to write operation is sent to the client
after the write is completed in both the
memory cache and the server’s disk
Write is performed only in the memory cache: It is written to the disk
only when commit operation is received for the file
Overcomes the performance bottleneck in the write-through
operation at the disk

Sukkur IBA University , Sukkur DS - Spring 2018 24

Sun Network File System
 Client caching
Client module caches the results of most of the operations to reduce
the number of requests transmitted to the servers
Writes by clients do not immediately propagate to the file copies
cached by other clients
Clients need to validate their cached data
Timestamp-based method is used for validation of cached data
Each data or meta data item in cache is tagged with two timestamps:
Tc – Time when the cache entry was last validated
Tm – Time when the block was last modified at the server
A cache entry is valid at time T if:
(T - Tc) is less than a freshness interval t OR
Tm at the client matches the value of Tm at the server
((T-Tc) < t) v (Tm client = Tm server)
The selection of t is a compromise between consistency and efficiency
A validity check is performed whenever a cache entry is used
If the first half of validity condition is false, Tm server is obtained by
getattr call to the server

Sukkur IBA University , Sukkur DS - Spring 2018 25

Sun Network File System
 Measures to reduce the traffic of getattr to the server
Whenever a new value of Tmserver is received, it is applied to all cache
entries derived from the relevant file
The current attribute values are sent ‘piggybacked’ with the results of
every operation on a file
Adaptive algorithm is used for setting freshness interval t
 Writes are handled by using a write-back strategy
 Bio-daemons are used to implement read-ahead and delayed write
Client module need not block waiting for reads to return or
writes to commit at the server
 Other Optimizations
The UDP packet size for Sun RPC is extended from the standard
UNIX version
Later versions of Sun RPC allow to negotiate for larger sized file
blocks to be exchanged between client and server
‘Piggybacking’ is used wherever possible

Sukkur IBA University , Sukkur DS - Spring 2018 26

Sun Network File System
 Performance
Initial measurements indicate that NFS does not impose performance
penalty in comparison with access to files stored on local disks
 Problem areas for performance:
Frequent use of getattr call to get timestamps for cache validation
Relatively poor performance for writes because of write-through
strategy used at the server
Commit mechanism improved performance
 NFS Summary
 Sun NFS provides good location and access transparency if the NFS
mount service is used properly
 NFS supports heterogeneous hardware and operating systems
 Server implementation is stateless enabling crash recovery without
any specific recovery procedures
 Migration of files or filesystem is not supported – manual intervention
for reconfiguration is allowed
Read-only file stores can be replicated on several NFS servers

 The Andrew File System - Reading Assignment

Sukkur IBA University , Sukkur DS - Spring 2018 27

Enhancements and further developments
 Describe the advances that enhance the performance, availability and
scalability of conventional distributed file systems
 NFS enhancements
Improve the one-copy update semantics by extending the NFS
protocol to include open and close operations and adding a callback
mechanism - enables the server to notify clients for cache invalidation
Spritely NFS: Protocol that added open and close calls with parameters
Parameters include mode and counts of the number of local
processes that currently have the file open
Server sends a callback to any client that is writing to cease
caching in case of a file opened in read mode
 It is possible to achieve one-copy update semantics without much
loss of performance
Adds extra implementation complexity and there is a need for a
recovery mechanism in case of a server failure
 Not Quite NFS (NQNFS): Improve performance with better use of
caching
Lease: The server sets an upper bound on the time for which a client
may hold an open file - helps in server crash recovery
Lease should be renewed to hold the file for a longer time

Sukkur IBA University , Sukkur DS - Spring 2018 28

Enhancements and further developments
 WebNFS
Enable web browsers and other internet applications to access files
that are ‘published’ using public file handle directly from NFS server
Avoids overheads associated with the emulation of UNIX file
operations included in the standard NFS clients
 Further enhancements and improvements in future versions of NFS
would include:
Features of WebNFS and NQNFS
On-the-fly recovery from server faults
Improve scalability by adding proxy servers
 Improvements in the storage organizations
Progress has been made in the organization of file data stored on
disks
Redundant Arrays of Inexpensive Disks (RAIDs)
Provides redundancy through error-correcting codes
Gives improved performance due to striping
Log-structured File Storage (LFS)
Accumulate a set of writes in memory and then commit them to disk in
large, contiguous, fixed-sized segments – log segments

Sukkur IBA University , Sukkur DS - Spring 2018 29

Enhancements and further developments
 New design approaches
Availability of high-performance switched networks have helped to
Provide persistent storage systems that distribute file data in a
highly scalable and fault-tolerant manner among many nodes on an
intranet
Separate the responsibility of reading and writing data from
responsibility of managing the metadata and servicing client
requests
 Approaches scale better than centralized servers
Demand a high level of trust among the computers that
cooperate to provide the service
xFS: A server less network file system architecture
Motivated by three factors
1. Use of multiple file servers so that fast switched LANs can be used
for transfer of bulk data to clients concurrently
2. Increased demand for access to shared data
3. Limitations of systems based on central file servers
 Initial evaluation showed a marked improvement in performance
over the existing NFS and AFS filesystems

Sukkur IBA University , Sukkur DS - Spring 2018 30