Linux Intermediate

ITS Research Computing Center C. D. Poon, Ph.D. Email: [email protected]

Class Material
Point web browser to
http://its.unc.edu/Research

Click on Training on the left column Click on ITS Research Computing Training
Presentations

Click on Linux Intermediate Commands,

Tips and Tricks

its.unc.edu

Outline
Linux Command Category Stdout/Stdin/Stderr, Pipe and Redirection,
Wildcards

Linux Command Review Break Tips and Tricks Conclusion Question and Exercise
its.unc.edu

Linux Command Category

its.unc.edu

Linux Command Category

Communication
ssh scp File/Directory Management cat cd chmod cp ln ls mkdir more less mv pwd dirs rm head tail wc Comparisons diff
its.unc.edu

Linux Command Category Contd

Searching
grep find locate

Archiving
compress uncompress gzip gunzip zcat tar

Text Processing
cut paste sort sed awk uniq
its.unc.edu

Linux Command Category Contd

System Status
chgrp chown date df du env who w uptime

Miscellaneous
bc cal clear man

its.unc.edu

Stdout/Stdin/Stderr Pipe and Redirection Wildcards

its.unc.edu

stdout stdin stderr

Output from commands
usually written to the screen referred to as standard output (stdout)

Input for commands

usually come from the keyboard (if no arguments are
given referred to as standard input (stdin)

Error messages from processes

usually written to the screen referred to as standard error (stderr)
its.unc.edu

Pipe and Redirection

Pipe (|): stdout of one command to stdin
of another command
command to a file

Output Redirection (>): stdout of a Output Appending (>>): stdout of a

command appending to a file from a file

Input Redirection (<): stdin of a command

its.unc.edu

10

Stderr Redirection
For tcsh &> filename

For bash
2>&1 filename

its.unc.edu

11

Wildcards
Multiple filenames can be specified using special
pattern-matching characters. The rules are:

? matches any single character in that position in

the filename

* matches zero or more characters in the filename. [] Characters enclosed in square brackets match
any name that has one of those characters in that position

Note that the UNIX shell performs these expansions

before the command is executed.

its.unc.edu

12

Linux Command Review

its.unc.edu

13

ssh
Log on to remote machine from Linux
ssh [email protected] ssh killdevil.unc.edu l cdpoon ssh killdevil ssh X kure.unc.edu ssh Y kure.unc.edu

its.unc.edu

14

ssh using SecureCRT in Windows

Using ssh, login to killdevil.unc.edu To start ssh using SecureCRT in Windows, do the following.
Start -> Programs -> Remote Services -> SecureCRT Click the Quick Connect icon at the top. Hostname: killdevil.unc.edu Login with your ONYEN and password

its.unc.edu

15

scp
Copy files and directories to and from remote computers
scp file1 killdevil.unc.edu:/nas02/home/cdpoon/. scp zircon.its.unc.edu:/home/cdpoon/file2 . scp r dir1 killdevil.unc.edu:/netscr/cdpoon/. scp r killdevil.unc.edu:/netscr/cdpoon/dir2 dir3 scp kure:/netscr/cdpoon/f killdevil:/nas02/home/cdpoon/.

its.unc.edu

16

cat
Read one or more files and print the on stdout
cat file1 cat file1 file2 file3 > file_all

cat file4 >> file_all

Append file4 to file_all

cat > file5

Create file at stdin, end with EOF (cntl-d normally, use stty a to find out)

cat > file6 << STOP

Create file at stdin, end with STOP
its.unc.edu

17

cd
Change directory, build-in shell command
cd /nas02/home/c/d/cdpoon cd ../../ cd ..
Change directory to 2 levels up Change directory to 1 level up

cd ~

Change directory to Home

cd
cd

Change directory to Home

Change to previous directory

its.unc.edu

18

chmod
Change the access mode of one or more files
chmod u+x file1 chmod go-w file2 chmod u=rwx, g=rx, o=x file3 chmod 751 file3 chmod =r file4 chmod 444 file4
Same as above, 4=r, 2=w, 1=x Same as above, 7=rwx, 5=rx, 1=x

its.unc.edu

19

cp
Copy a file/dir to another file/dir
cp file1 file2 cp file1 ../dir/file2 cp file1 ../dir/. cp r dir1 dir2
Copy to the same directory and change filename Copy to different directory and change filename Keep the same filename

Copy directory recursively Copy directory recursively to another directory

cp r dir1 new_dir/dir2

cp p file3 file4

Preserve the modification time and permission modes

its.unc.edu

20

ln
Create links for file/dir and allow them to be accessed by different names
ln file1 file2 ln dir1 dir2 ln s dir1 dir2 ln s file3 file4 ln s dir/file5 file6
Hard link for file Hard link not allowed for directory

Soft link for directory, dir2 -> dir1 Soft link, file4 -> file3 Soft link, file6 -> dir/file5

its.unc.edu

21

ls
List all files and directories in the current directory
ls

ls a
ls l ls lh

List files/directories starting with . too

Long listing

List file sizes in human readable format and long list format

ls F

Flag filenames by appending / to directories, * to executables files,

and @ to symbolic links

its.unc.edu

22

mkdir
Create one of more directories
mkdir dir1 mkdir p dir1/dir2/dir3
Create intervening parent directories if they dont exist Same as mkdir dir1; cd dir1; mkdir dir2; cd dir2; mkdir dir3; cd ../../

its.unc.edu

23

more
Display files on a terminal, one screen at a time
more file1
Hit space bar for another page, q to quit

more d file2
more c file3

Display the prompt Press space to continue, q to quit

Page through the file by clearing each window instead of scrolling

its.unc.edu

24

less
Works like more but allows backward and forward movement
less file1
Hit space bar for another page, q to quit

Hit b to scroll backward one window

Hit /pattern to highlight pattern in the text Hit Return to scroll one line at a time

its.unc.edu

25

mv
Move files and directories within the same machine and/or rename them
mv file1 dir1/file1 mv file3 file4 mv dir1 dir2
Move file1 to dir1, Same as mv file1 dir1/ Rename file3 to file4 Rename directory dir1 to dir2

mv dir3 dir4/dir5/dir6
Rename directory dir3 to dir6 and move to dir4/dir5 directory

its.unc.edu

26

pwd dirs
Print the full pathname of the current directory
pwd dirs
like pwd C shell and bash shell built-in command, works Print working directory in long listing

dirs l

its.unc.edu

27

rm
Delete one or more files and directories Delete empty directory with rmdir
rm file1 rm file* rm i file* rm r dir1
Remove all files with filename starting as file

Prompt for y (remove the file) or n (do not remove the file)

Delete directory dir1 and its content

its.unc.edu

28

head tail
Print first/last few lines of one or more files
head file1 head n100 file2 tail file*
Print the first 10 lines of file file1 Print the first 100 lines of file file2
Print the last 10 lines of files with filename starting as file Print the last 10 lines of file file3 and follow file as it grows

tail f file3

its.unc.edu

29

wc
Print a character, word, and line count for files
wc c file1 wc l file2 wc w file3
Print character count for file file1

Print line count for file file2

Print word count for file file3

its.unc.edu

30

diff
Report lines that differ between file1 and file2, with file1 text flagged by < and file2 by >
diff file1 file2
Show difference between file1 and file2

its.unc.edu

31

grep
Search for lines that match a regular expression
grep abc file1
Print line(s) in file file1 with abc

grep i abc file2

Print line(s) in file file2 with abc ignoring uppercase and lowercase distinctions

its.unc.edu

32

find
Find particular groups of files
find . name temp
Find file named temp in current directory

find /etc name rc*

Find file(s) in /etc directory with name starting with rc

find /usr/share/man type d name man*

Find directories in /usr/share/man with name starting with man

its.unc.edu

33

locate
Find files with matching pattern in database prepared by updatedb, Database needed to be updated daily
locate which locate c which
Find files named with pattern which in the OS Count number of files named with pattern which in the OS Find files named with pattern which in the OS ignoring case distinctions

locate i which

its.unc.edu

34

compress uncompress
Reduce or expand the size of one or more files using adaptive Lempel-Ziv coding Use uncompress to expand data
compress file1 compress f file2
create file2.Z Reduce the size of file1 and create new file named file1.Z Force to reduce the size of file2 and new file named

uncompress file3.Z
its.unc.edu

Expand file3.Z and restore file3

35

gzip gunzip
Reduce or expand the size of one or more files using Lempel-Ziv coding (LZ77) Use gunzip to expand data
gzip file1
Reduce the size of file1 and create new file named file1.gz

gzip f file2

Force to reduce the size of file2 and create new file named file2.gz

gunzip file3.gz

Expand file3.gz and restore file3

its.unc.edu

36

zcat
Expand the size of one or more files created by compress or gunzip List file contents to stdout without deleting the .Z or .gz file
zcat file1.Z
Expand file1.Z and list the content of file1 in stdout

zcat file2.gz

Expand file2.gz and list the content of file2 in stdout

its.unc.edu

37

tar
Archive files and directories
Create a single file with extension .tar
tar cvf file123.tar file1 file2 file3
Create archive file named file123.tar in verbose mode with contents, file1, file2, and file3

tar xvf file123.tar

Expand file123.tar in verbose mode and generate the original files and directories back

its.unc.edu

38

cut
Remove sections from each line of files
cut d: -f1,5 /etc/passwd
Use field delimiter : to locate fields 1 and 5 from file /etc/passwd to extract usernames and real names

cut c4 file1
Take character 4 out from each line of file1 and display in stdout

its.unc.edu

39

paste
Merge lines of files
$ cat file1
1 2 $ cat file2 a b $ paste file1 file2 1 2 a b

c
$ paste s file1 file2 1 2 a b c

its.unc.edu

40

sort
Sort lines of text files
sort fd file1
Alphabetize lines (-d) in file1 and ignore lower and upper cases (-f)

sort t: -k3 -n /etc/passwd

Take column 3 of file /etc/passwd separated by : and sort in arithmetic order

its.unc.edu

41

sed
Edit one or more files without user interaction using stream editor
sed s/xx/yy/g file1
Substitude all occurrences of xx in file1 with yy and display on stdout

sed /abc/d file1

Delete all lines containing abc in file1

sed /BEGIN/,/END/s/abc/123/g file1

Substitute XYZ on lines between BEGIN and END with xyz in file1

its.unc.edu

42

awk
Process files by pattern-matching
awk F: {print $1} /etc/passwd
Extract the 1st field separated by : in /etc/passwd and print to stdout

awk /abcde/ file1

Print all lines containing abcde in file1

awk /xyz/{++i}; END{print i} file2

Find pattern xyz in file2 and count the number

awk length <= 1 file3

Display lines in file3 with only 1 or no character

its.unc.edu

43

uniq
Report or omit repeated lines
uniq file1
Filter adjacent matching lines from file named file1 , writing to stdout

uniq c file1
Prefix lines by the number of occurrences from file named file1

its.unc.edu

44

chgrp
Change the group ownership of one or more files or directories
chgrp employee file1
Change group ownership to employee for file file1

chgrp R student dir1

Change group ownership to student for directory dir1 including subdirectories recursively

its.unc.edu

45

chown
Change the ownership of one or more files or directories
chown employee file1
Change ownership to employee for file file1

chown R student dir1

Change ownership to student for directory dir1 including subdirectories recursively

its.unc.edu

46

date
Print the current date and time in certain format Set the current date and time
date
Print the current date and time

date +%D
Print the current date and time in mm/dd/yy format

date 1201160108
Set the current date and time to Dec 01 4:01pm 2008

date d fri
Show the date of the coming Friday

its.unc.edu

47

df
Report the number of used and free disk block on all mounted file systems
df
Print used and free disk block on all mounted file system

df -h
Print used and free disk block in human readable format

df -k
Print used and free disk block in kb

its.unc.edu

48

du
Print disk usage of directories and its subdirectories
du dir1
Print disk usage in kilobyte of directory dir1

du -block-size=1M dir2
Print disk usage in megabyte of directory dir2

du hs dir3
format of

Print summarized disk usage in human-readable directory dir3

its.unc.edu

49

env
Display the current environment variables or set new values
env
Display all of the current environment variables

its.unc.edu

50

who
Display information about the current status of the system
who
who b
Display the names of users currently logged in to the system
Report information about the last reboot

who am I

Print the username of the invoking user

its.unc.edu

51

w
Print summaries of system usage, currently logged-in users, and what they are doing
w w s
Print summaries of system usage, currently logged-in users

Display in short form

its.unc.edu

52

uptime
Print the current time, amount of time logged in, and the system load averages
uptime
Print a one line display of the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, 15 minutes

its.unc.edu

53

bc
Interactively perform arbitrary-precision arithmetic or convert numbers from one base to another, type quit to exit
bc
Invoke bc

1+2
5*6/7 ibase=8 20 16 ibase=10

Evaluate an addition
Evaluate a multiplication and division Change to octal input Evaluate this octal number Output decimal value Change back to decimal input

its.unc.edu

54

cal
Print calendar of a month or all months in a year
cal cal 2 2009 cal 2009 cal -3
Print calendar of the current month Print calendar of February 2009 Print calendar of all months in 2009 Display previous/current/next months

its.unc.edu

55

clear
Clear the terminal display and have the prompt locate at the top of the terminal window
clear
Clean up the current terminal display

its.unc.edu

56

man
Display information from the online reference manuals
man man man k link compile
Display the manual for the command man

Display commands related to linking and compiling using a keyword search

its.unc.edu

57

Break

its.unc.edu

58

Tips and Tricks

its.unc.edu

59

Tips and Tricks #1

Show files changed on a certain date in all directories
ls l * | grep Sep 26
Show long listing of file(s) modified on Sep 26

ls lt * | grep Dec 18 | awk {print $9}

Show only the filename(s) of file(s) modifed on Dec 18

its.unc.edu

60

Tips and Tricks #2

Sort files and directories from smallest to biggest or the other way around
du k s * | sort n
Sort files and directories from smallest to biggest

du ks * | sort nr
Sort files and directories from biggest to smallest

its.unc.edu

61

Tips and Tricks #3

Change timestamp of a file
touch file1
If file file1 does not exist, create it, if it does, change the timestamp of it

touch t 200902111200 file2

Change the time stamp of file file2 to 2/11/2009 12:00

its.unc.edu

62

Tips and Tricks #4

Find out what is using memory
ps ely | awk {print $8,$13} | sort k1 nr | more

its.unc.edu

63

Tips and Tricks #5

Remove the content of a file without eliminating it
cat /dev/null > file1

its.unc.edu

64

Tips and Tricks #6

Backup selective files in a directory
ls a > backup.filelist
Create a file list

vi backup.filelist
Adjust file backup.filelist to leave only filenames of the files to be backup

tar cvf archive.tar `cat backup.filelist`

Create tar archive archive.tar, use backtics in the cat command

its.unc.edu

65

Tips and Tricks #7

Get screen shots
xwd out screen_shot.wd Invoke X utility xwd, click on a window to save the image as screen_shot.wd display screen_shot.wd Use ImageMagick command display to view the image screen_shot.wd Right click on the mouse to bring up menu, select Save to save the image to other formats, such as jpg.

its.unc.edu

66

Tips and Tricks #8

Sleep for 5 minutes, then pop up a message Wake Up
(sleep 300; xmessage near Wake Up) &

its.unc.edu

67

Tips and Tricks #9

Count number of lines in a file
cat /etc/passwd > temp; cat temp | wc l; rm temp wc l /etc/passwd

its.unc.edu

68

Tips and Tricks #10

Create gzipped tar archive for some files in a directory
find . name *.txt | tar c T - | gzip > a.tar.gz
find . name *.txt | tar cz T - -f a.tar.gz

its.unc.edu

69

Tips and Tricks #11

Find name and version of Linux distribution, obtain kernel level
uname -a
head n1 /etc/issue

its.unc.edu

70

Tips and Tricks #12

Show system last reboot
last reboot | head n1 who -b

its.unc.edu

71

Tips and Tricks #13

Combine multiple text files into a single file
cat file1 file2 file3 > file123 cat file1 file2 file3 >> old_file cat `find . name *.out` > file.all.out

its.unc.edu

72

Tips and Tricks #14

Create man page in pdf format
man t man | ps2pdf - > man.pdf acroread man.pdf

its.unc.edu

73

Tips and Tricks #15

Remove empty line(s) from a text file
awk NF>0 < file.txt
Print out the line(s) if the number of fields (NF) in a line in file
file.txt is greater than zero

awk NF>0 < file.txt > new_file.txt

Write out the line(s) to file new_file.txt if the number of fields (NF) in a line in file file.txt is greater than zero

its.unc.edu

74

Conclusion

its.unc.edu

75

Conclusion
Many ways to do a certain thing Unlimited possibilities to combine commands
with |, >, <, and >> shell script

Even more powerful to put commands in

Slightly different commands in different

Linux distributions

Emphasized in System V, different in BSD

its.unc.edu

76

Questions ?
Exercise

its.unc.edu

77