Unix _linux_fundamentals_for_hpc-_b

UNIX & LINUX Fundamentals for
HPC
A Short-Series Presentation

Purpose of This Lecture
• Fundamentals of using UNIX and UNIX-like
systems
• History of UNIX & Linux
• Basic system commands
• Data Management
• Constructing basic shell scripts

History of UNIX
• "...the number of UNIX installations has grown to 10, with more
expected..."
– Dennis Ritchie and Ken Thompson, June 1972
• "... When BTL withdrew from the project, they needed to rewrite
an operating system (OS) in order to play space war on another
smaller machine (a DEC PDP-7 [Programmed Data Processor]
with 4K memory for user programs). The result was a system
which a punning colleague called UNICS (UNiplexed
Information and Computing Service)--an 'emasculated Multics';
no one recalls whose idea the change to UNIX was"

Bringing UNIX to the desktop
• Unix was very expensive
• Microsoft DOS was the mainstream OS
• MINIX, tried but was not a full port
• An opensource solution was needed!

Linux 0.02 – October 5, 1991
• “Do you pine for the nice days of minix-1.1, when men were
men and wrote their own device drivers?
Are you without a nice project and just dying to cut your teeth on
a OS you can try to modify for your
needs? Are you finding it frustrating when everything works on
minix? No more all-nighters to get a nifty program working?
Then this post might be just for you :-)” - Linus Benedict
Torvalds
• "I still maintain the point that designing a monolithic kernel in
1991 is a fundamental error. Be thankful you are not my student.
You would not get a high grade for such a design :-)"
(Andrew Tanenbaum to Linus Torvalds)

1990’s Movers and Shakers
• Richard Stallman, father of the GNU Project
• Linus Torvalds

Why UNIX/LINUX is Still Used
• 30+ years of development
• Many academic, scientific, and system tools
• Open Source
• System Stability
• Lightweight
• Easy Development

The Basics
• Command-Line
– Interaction with UNIX & LINUX is based on
entering commands to a text terminal
– Oftentimes there are no ‘warnings’ with
commands, no ‘undo’
• The Shell
– The user environment that enables interaction with
the kernel, or lower-system OS.
– Internet Explorer would be a shell for Microsoft
Windows.

Common Shells
• sh – the original UNIX shell; still located in /bin/sh
• Bash is a Unix shell written for the GNU Project and
is installed on most Linux systems.
• csh – C Shell, modeled after the C programming
language used by UNIX systems
• tcsh – C Shell with modern improvements such as
filename completion
• echo $SHELL – displays what shell your account is
using
• chsh – change your shell

Before we go further…
• Read the Manual.
– man command
– man [section] command
– man –k keyword (search all manuals based on keyword)
• Most commands have a built-in UNIX manual, even
the man command!
• Commands without manuals have help too, with –h,
--help, or /? option.

The Manual
• The manual pages are divided into eight sections
depending on type of command.
– 1 commands and applications
– 2 system calls
– 3 C library functions
– 4 special files
– 5 file formats
– 6 games
– 7 misc.
– 8 system administration utilities

Why sections are important
lslogin2$ man 2 time
TIME(2) Linux Programmer's
Manual TIME(2)
NAME
time - get time in seconds
SYNOPSIS
#include <time.h>
time_t time(time_t *t);
lslogin2$ man 1 time
TIME(1)
TIME(1)
NAME
time - time a simple command or give
resource usage
SYNOPSIS
time [options] command [arguments...]
DESCRIPTION
The time command runs the specified
program command with the given arguments.
When command finishes, time writes a
message to standard output giving timing

Conventions for this lecture
• This lecture is too short to give you all of the
options, so look at the manual for specific
syntax for commands.
• Commands will be in bold, options will be in
italics.
– command -arguments
• Output will be shown in its own bordered
table

Command Conventions
• In helpfiles and manuals, commands will
have required input and option input
• cp [OPTION] source destination
– Optional arguments are in brackets, require
arguments are not.
• cp –R or cp - -recursive
– Short options ‘-’, long options ‘- -’

Directories
• What is a working directory?
– The directory your shell is currently associated
with. At anytime in the system your login is
associated with a directory
• pwd – view the path of your working directory
• ls – view your working directory

Who’s Path is it Anyway?
• UNIX treats the directory structure as a
hierarchy of individual paths
/
usrhomedev
david bin
/ (root directory)
/home
/home/david

Finding your home
• Each user has a home directory which can be found
with:
– cd
– cd ~david
– cd $HOME
• The tilde character ‘~’ will tell the shell to auto-
complete the path statement for the cd command
• $HOME refers to an environment variable which
contains the path for home.

More file commands
• cd directory - change your current working directory
to the new path
• ls –a – show hidden files
– Hidden files are files that begin with a period in the filename
‘.’
• mv - moves one file to another
• cp – copies files or directories
• rm – remove files & directories
– rm –rf – remove everything with no warnings
– rm –rf * - most dangerous command you can run!

Recursive Directories
• Oftentimes a manual will refer to ‘recursive’
actions on directories. This means to perform
an action on the given directory and
recursively to all subdirectories.
– cp –R source destination – copy recursively all
directories under source to destination

The Bit Bucket
• /dev/null
– Throw items away into /dev/null and they will be
gone forever. Good place to redirect trash output
to.
• Other interesting files on the system
– /dev/rand
• Pseudo random number generator
– /dev/zero
• Fill zeros. Very fast ,use with caution otherwise you may
get nasty email from administrator.

Relative vs.. Absolute Path
• Commands expect you to give them a path to a file. Most
commands will let you provide a file with a relative path, or a
path relative to your working directory.
– ../directory - the ‘..’ refers to looking at our previous directory first
– ./executable - ‘.’ says this directory, or our working directory
• Absolute, or Full paths are complete. An easy way to know if a
path is absolute is does it contain the ‘/’ character at the
beginning?
– /home/user/directory/executable - a full path to file executable

Poking around in $HOME
• How much space do I have?
– quota – command to see all quotas for your
directories are, if any.
• How much space am I taking up?
– du - command to find out how much space a
folder or directory uses.
– df – display space information for the entire
system

Helpful Hints on Space
• Almost all commands that deal with file space
will display information in Kilobytes, or Bytes.
Nobody finds this useful.
• Many commands will support a ‘-h’ option for
“Human Readable” formatting.
– ls –lh - displays the working directory files with a
long listing format, using “human readable”
notation for space

Representing Space
• Bit – either a 1 or 0
• Byte – 8 bits “0000 1111”, or “x0F”, or 16
• KB, Kilobyte - 1024 Bytes
• MB, Megabyte – 1024KB
• GB, Gigabyte – 1024MB
• TB, Terabyte – 1024 GB
• PB, Petabyte – 1024 TB

Permissions
• The *NIX systems are multi-user environments
where many users run programs and share data.
Files and directories have three levels of
permissions: World, Group, and User.
• The types of permissions a file can contain are:
xwr
Execute PermissionsWrite PermissionsRead permissions

Permissions Cont.
r--r--rw-
Others (everyone else)GroupUser (owner)
• File permissions are arranged in three groups
of three characters.
• In this example the owner can read & write a
file, while others have read access

Changing Permissions
• chmod – change permissions on a file or
directory
• chgrp and chown – change group ownership
to another group (only the superuser can
change the owner)
– Both options support ‘-R’ for recursion.

All About Me
• Every userid corresponds to a unique user or system process
– whoami – returns the userid of the current user
– passwd – change password
• What is my group? – support!
lslogin2$ ls -l ~eturner
total 391632
drwx------ 2 eturner support 4096 Sep 12 15:41 anothertest
-rwx------ 1 eturner support 8487 Jun 7 13:12 a.out
drwx------ 5 eturner support 4096 Apr 7 2006 automake-test

Everyone else
• who – show all other users logged in
• finger – show detailed information about a user
lslogin2$ finger eturner
Login: eturner Name: Evan Turner
Directory: /home/utexas/staff/eturner Shell: /bin/bash
On since Thu Sep 20 15:52 (CDT) on pts/37 from 129.114.50.161
Mail last read Thu Sep 13 15:14 2007 (CDT)
Plan:
User Support
Texas Advanced Computing Center
eturner@tacc.utexas.edu

What everyone else is up to
• top – show a detailed, refreshed, description of
running processes on a system.
• uptime – show the system load and how long the
system has been up.
– ‘load’ is a number based on utility of the cpu’s of the
system. A load of 1 indicates full load for one cpu.
lslogin2$ uptime
15:43:52 up 3 days, 20:45, 154 users, load average: 2.04, 2.00, 2.36

Working With Programs
• Commands or programs on the system are
identified by their filename and by a process
ID which is a unique identifier.
– ps – display process information on the system
– kill pid – terminates the process id
– ^c (control+c) terminates the running program
– ^d (control+d) terminates your session.
• Only you and the superuser (root) has
permissions to kill processes you own.

Advanced Program Options
• Oftentimes we must run a command in the
background with the ampersand ‘&’ character
– command –options & - runs command in
background, prompt returns immediately
• Match zero or more characters wildcard ‘*’
– cp * destination - copy everything to destination
– This option can get you into trouble if misused

Editing Files
• emacs vs. vi
– Among the largest ‘nerd battle’ in history. emacs relies
heavily on key-cords (multiple key strokes), while vi is mode
based. (editor mode vs. command mode)
– vi users tend to enter and exit the editor repeatedly, and use
the Unix shell for complex tasks, whereas emacs users
usually remain within the editor and use emacs itself for
complex tasks
• pico (nano)
– editor originally used for the email client pine, simple no-frills
editor which resembles notepad for windows.

Input and Output
• Programs and commands can contain an input and
output. These are called ‘streams’. UNIX
programming is oftentimes stream based.
– Programs also have an error output. We will see later how to
catch the error output.
• STDIN – ‘standard input,’ or input from the keyboard
• STDOUT – ‘standard output,’ or output to the screen
• STDERR – ‘standard error,’ error output which is sent
to the screen.

File Redirection
• Oftentimes we want to save output (stdout)
from a program to a file. This can be done
with the ‘redirection’ operator.
– myprogram > myfile – using the ‘>’ operator we
redirect the output from myprogram to file myfile
• Similarly, we can append the output to a file
instead of rewriting it with a double ‘>>’
– myprogram >> myfile – using the ‘>’ operator we
append the output from myprogram to file myfile

Input Redirection
• Input can also be given to a command from a
file instead of typing it to the screen, which
would be impractical.
– cat programinput > mycommand - This command
series starts with the command ‘cat’ which prints a
file to a screen. programinput is printed to stdout,
which is redirected to a command mycommand.

Redirecting stderr
• Performing a normal redirection will not
redirect sdterr. In Bash, this can be
accomplished with ‘2>’
– command 2> file1
• Or, one can merge stderr to stdout (most
popular) with ‘2>&1’
– command > file 2>&1

Pipes
• Using a pipe operator ‘|’ commands can be
linked together. The pipe will link the
standard output from one command to the
standard input of another.
• Very helpful for searching files

Searching
• A large majority of activity on UNIX systems involve
searching for files and information.
– find – utility to find files
– grep – the best utility ever written for UNIX, searches for
patterns inside files and will return the line, if found
lslogin2$ find ./ -name output-evansapp-n0.out534086
./test/run_0/output-evansapp-n0.out534086
lslogin2$ cat test/run_0/output-evansapp-n0.out534086 | grep CPU
CPU time : 0.03 sec.

Packing Files
• When creating backups of files, or
transferring to other hosts, files must be
packed into larger files. This is needed for
ease of manipulation, transfer speeds, and
file management.
• tar – create or extract a packed file. tar
stands for ‘tape archive’.

Compressing Files
• Compressing files can gain file space at the
expense of cpu time to compress and
decompress files.
• Compression works well for text files, but not
as well for binary files with random data such
as float values.
• Compression algorithms & commands
– gzip, gunzip, bzip2, bunzip2

Using tar to create compressed files
• Tar will create compressed files for you
– tar –czvf mytarfile.tar.gz directory – creates a
compressed file named mytarfile.tar.gz containing
all of the files in the directory directory
– tar –xzvf mytarfile.tar.gz – uncompresses all
directories and files inside the file mytarfile.tar.gz
into the working directory

Testing Compression
• Using an example from a dataset of visual MRI binary
data that is used for an application called Freesurfer
three different compression methods will be tested.
The dataset contains a mix of binary and text data.
– The collected data set which includes 128 individual 180KB
binary files which is 42MB in uncompressed form.
lslogin2$ du -h 014-anon/
13M 014-anon/Deface
180K 014-anon/002
42M 014-anon/
lslogin2$

The dataset is compressed using bzip,
gzip, and compress (Z)
lslogin2$ time tar -cjf ziptest.tbz 014-anon/
real 0m49.410s
user 0m7.840s
sys 0m0.921s
lslogin2$ ls -lth ziptest.tbz -rw------- 1 eturner support 23M Sep 12 15:52 ziptest.tbz
lslogin2$ time tar -cZf bziptest.tar.Z 014-anon/
real 0m51.255s
user 0m1.306s
sys 0m2.445s
lslogin2$ ls -lth ziptest.tar.Z -rw------- 1 eturner support 34M Sep 12 15:55 ziptest.tar.Z
lslogin2$ time tar -czf ziptest.tar.gz 014-anon/
real 0m46.949s
user 0m3.116s
sys 0m1.046s
lslogin2$ ls -lth ziptest.tar.gz -rw------- 1 eturner support 26M Sep 12 15:57 ziptest.tar.gz
lslogin2$

Results
• The test shows that bzip compression is the
most efficient at the expense of the most CPU
time. gzip is generally a good all-around
compression algorithm because it gives
decent performance with an average CPU
load.
• It is a good idea to test your own dataset.

Connecting to Another Machine
• Secure Shell vs. Restricted Shell
– ssh is an encrypted remote login program that is “secure” to
trust across non secure networks.
– rsh is a non-encrypted version of ssh that is used only
between sites that are inside secure networks.
– rsh provides faster file transfer speeds. However, most
TACC systems do not allow rsh connections, even from
another TACC system.
– rsh and ssh take similar arguments
• ssh userid@hostname
• rsh userid@hostname

Copying Files to Remote Hosts
• copy local file lfile to rfile on remote machine
rsys
– scp lfile rsys:rfile
• -p preserves modification time, access time and mode
from original
– scp -p lfile rsys:rfile
• copy rfile from remote machine rsys to local
file lfile
– scp -p rsys:rfile lfile

Running Commands on a Remote Host
• Commands can be executed on a remote
host with ssh
• Don’t set up a passphrase, as it will interfere
with job submission
• ssh userid@hostname “ls” - run ls on remote
host hostname

Advanced Movement
gtar -czf – foldertoarchive | bzip2 | ssh archive.tacc.utexas.edu "cat >
${ARCHIVE}/myfile.tar.bz2“
• This statement creates a bzip tar file and
sends it to the remote host
archive.tacc.utexas.edu
• What is nifty about this…
– Since the command is ‘inline’, no local backup
copy is created on the local host.

Basic Shell Scripts
• Many times it is helpful to create a ‘script’ of
commands to run instead of typing them in
individually. Scripts can be made to aid in post-
processing, system administration, and automate
menial tasks
• #!/bin/bash
– First statement inside a script, will list which shell to run this
script in
• # - says what will follow is a comment and not to
execute

Variables
• By convention system variables are capitalized
– HOME – location of the home directory
– OLDPWD – location of the previous working directory
– PATH – locations to look inside for executable files
• Setting system variables differs by shell. bash uses
export, csh uses setenv
• User defined variables in scripts are lower-case by
convention
– myvariable=10 – sets myvariable to 10
– echo $myvariable - prints myvariable

My Environment
• View all system variables by the command
‘env’
• Depending on shell, startup commands can
be managed with the files .profile for bash
and .cshrc with c shell

Conditionals
if condition
then
condition is zero (true - 0) execute all
commands up to else statement
else
if condition is not true then execute all
commands up to fi
fi

Multilevel Conditionals
if condition
then
condition is zero (true - 0) execute all commands
up to elif statement
elif condition1
then
condition1 is zero (true - 0) execute all commands up to
elif statement
elif condition2
then
condition2 is zero (true - 0) execute all commands up to
elif statement
else
None of the above condtion,condtion1,condtion2
are true (i.e. all of the above nonzero or false)
execute all commands up to fi
fi
fi

Performing Loops
• Loops are statements that are repeated until the conditions are
met.
for { variable name } in { list }
do
execute one for each item in the list until the list is not
finished (And repeat all statement between do and done)
done
for i in 1 2 3 4 5
do
echo "Welcome $i times“
done

While Loop
while [ condition ]
do
command1
command2
command3 .. ....
done

Putting it Together
#!/bin/bash
#my first script
#scp replacement
remotefile=labresults
myserver=home.utexas.edu
mylsinfo=`ssh $myserver -la $remotefile 2>&1`
ismissing=`echo $mylsinfo | grep ERROR`
if [ "$ismissing" ]
then
echo "$remotefile not found! Exiting!"
else
ssh -n "cat < $remotefile" > $localfile
fi

• mylsinfo=`ssh $myserver -la $remotefile
2>&1`
– `Backticks` are used to place output from a
command into a variable
• if [ "$ismissing" ]
– Is $ismissing set (has a value)? If so then the
expression is true, otherwise false

Archiving Tools
• Several utilities assist with transferring data to
the tape archive system; Ranch.
• bbcp (recommended)
• rls – remote ls
• tar
– tar cvf - <dirname> | ssh ranch "cat >
$ARCHIVE/<tarfile.tar>"

Remote LS
#!/bin/bash
myls=`(echo ${ARCHIVER} | grep archive >/dev/null && echo
"/etc/dmf/dmbase/bin/dmls") || (echo ${ARCHIVER} | grep ranch
>/dev/null && echo "/opt/SUNWsamfs/bin/sls")`
if [ "$myls" ]
then
rsh ${ARCHIVER} -n "$myls $*"
else
echo "archiver not found"
exit 1
fi

Babar File Copy
• The general syntax of bbcp:
– bbcp [options] <file or directory> <to Machine>:<relative
path>/<file or directory>
• To transfer <data> as <data>:
– bbcp <data> ${ARCHIVER}:
• To transfer <data> and force replacement:
– bbcp -f <data> ${ARCHIVER}:<data>
• To transfer directory <dir1> and subdirectories:
– bbcp -r <dir1> ${ARCHIVER}:

Modules
• Useful module commands:
module avail
module unload
module swap
• Troubleshooting:
– tacc-help module (Lonestar only)
• taccdiag – environment check for job submission
– check module dependencies, unload a competing
module before loading the replacement, or use
swap option:
module swap mvapich2 openmpi

Environment setup
• To automatically setup modules in startup
scripts:
– use the .login_user startup script, instead of
.cshrc_user
– use .profile_user for Bourne based shells

Grid Engine Usage Info
• Available only on Ranger.
• Use the qacct utility to get accounting data.
– Produces a summary of information for wall-clock
time, cpu-time, and system time for hostname,
queue-name, group name, owner, job-name, job-
ID, etc.
– See the qacct man page for more details.

UNIX vs.. Windows files
• File formats are different between the two
operating systems
• Use the UNIX command dos2unix to convert
files – especially script files - created on
Windows, so they will work on UNIX

Debugging Parallel Programs
• DDT available on Ranger and Lonestar
– interactive debugger for MPI applications
– GUI capable
• Not available on Stampede
• New Debugging Class available soon

References
• Graham Glass & King Ables Linux for
Programmers and Users (2006)
• Mark G. Sobell A Practical Guide to UNIX
System V1985
• Amir Afzal UNIX Unbounded A Beginning
Approach TE 2000
• http://www.english.uga.edu/hc/unixhistoryrev.
html
• https://netfiles.uiuc.edu/rhasan/linux/

Unix _linux_fundamentals_for_hpc-_b

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Unix _linux_fundamentals_for_hpc-_b (20)

Recently uploaded (20)

Unix _linux_fundamentals_for_hpc-_b