Sarfraaz Ahmed Linux Course

Basic LinuX &

Shell ScriptinG
— By : Sarfraaz Ahmed

Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.
 Linux Course

Introduction to ‘sed’

Sarfraaz Ahmed
 Linux Course

sed
• Stream Editor

• Non-interactive

• Uses commands of “ed” editor

• Loads a line without ‘\n’ into its pattern buffer

• Apply commands on the text in pattern buffer

• Display the result in stdout

Sarfraaz Ahmed

 Linux Course

sed Operations
• Print specific lines of a file

• Redirect this as input to another command

• Print lines between a start and end pattern string

• Ignore lines containing a specific pattern

• Search a replace text in a line

• Search a replace a specific or all occurrences of a

text

Sarfraaz Ahmed

 Linux Course

Printing Lines
Printing with p
command command

sed ‘p’ le.txt sed -n ‘p’ le.txt

line one line one
line one line two
line two ….
line last -n option
Each input line line two
suppresses
is echoed after …. the echoing
commands have
ben applied to it line last of input line
line last

Sarfraaz Ahmed
fi
fi
 Linux Course

Printing speci c lines

Printing with p

Print line Print last

number 2 line

sed -n ‘2p’ le.txt sed -n ‘$p’ le.txt

line two line last

Sarfraaz Ahmed
fi
fi
fi
 Linux Course

Printing Speci c lines

Printing with p
Print line Print
numbers line number
2 to 5 2 to last line

sed -n ‘2,5p’ le.txt sed -n ‘2,$p’ le.txt

line two line two
line three line three
line four ….
line ve line last

Sarfraaz Ahmed
fi
fi
fi
fi
 Linux Course

Printing speci c lines

Print Printing with p Print line
line number 2
number 2 and up to the
followed by 5 line that contains the
lines text “four”

sed -n ‘2,+5p’ le.txt sed -n ‘2,/four/’ le.txt

line two line two
line three line three
line four line four
line ve
line six
line seven

Sarfraaz Ahmed
fi
fi
fi
fi
 Linux Course

Printing lines
Printing with p

sed -n ‘2p’ le.txt Print line number 2

sed -n ‘2, 8p’ le.txt Print line numbers 2 to 8

sed -n ‘2, $p’ le.txt Print line number 2 to last line

Print line number 2 followed

sed -n ‘2, +4p’ le.txt
by the next 4 lines

Print line number 2 up to the

sed -n ‘2, /xxx/p’ le.txt
line that contains the text xxx

Sarfraaz Ahmed
fi
fi
fi
fi
fi
 Linux Course

Printing lines with pattern

Printing with p Print lines
that contain the
text xxx

sed -n ‘2p’ le.txt sed -n ‘/xxx/p’ le.txt

sed -n ‘2, 8p’ le.txt sed -n ‘/xxx/, 8p’ le.txt

sed -n ‘2, $p’ le.txt sed -n ‘/xxx/, $p’ le.txt

sed -n ‘2, +4p’ le.txt sed -n ‘/xxx/, +4p’ le.txt

sed -n ‘2, /xxx/p’ le.txt sed -n ‘/xxx/, /yyy/’ le.txt

Sarfraaz Ahmed
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
 Linux Course

Deleting Lines
Delete with d Delete lines
Delete line that contain the
number 2 text xxx

sed -n ‘2d’ le.txt sed -n ‘/xxx/d’ le.txt

sed -n ‘2, 8d’ le.txt sed -n ‘/xxx/, 8d’ le.txt

sed -n ‘2, $d’ le.txt sed -n ‘/xxx/, $d’ le.txt

sed -n ‘2, +4d’ le.txt sed -n ‘/xxx/, +4d’ le.txt

sed -n ‘2, /xxx/d’ le.txt sed -n ‘/xxx/, /yyy/d’ le.txt

Here, delete means, skip. Don’t show the lines. Original le is una ected

Sarfraaz Ahmed
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
ff
 Linux Course

Search and Replace

Search Replace Print the
for “line” with “point” output

sed -n ’s/line/point/p’ le.txt

point one
point two
point three
…
last point

Sarfraaz Ahmed
fi
 Linux Course

Search and Replace

In Speci c Lines only

sed -n ’2 s/line/point/p’ le.txt

Do this
point two
operation only for
line 2
sed -n ’2, +2 s/line/point/ip’ le.txt
point two
point three Search “line”
point four ignoring its case

sed -n ’2, /three/ s/line/point/p’ le.txt

point two
point three

Sarfraaz Ahmed
fi
fi
fi
fi
 Linux Course

Search and Replace

For Speci c occurrences

sed -n ’s/e/E/p’ le.txt

linE one
linE two
Replace
linE three only the 2nd
… occurrence in
last linE a line

sed -n ’s/e/E/2p’ le.txt

line onE
line thrEe
line vE

Sarfraaz Ahmed
fi
fi
fi
fi
 Linux Course

Search and Replace

All occurrences

sed -n ’s/e/E/gp’ le.txt

linE onE Replace all
linE two occurrences in
linE thrEE a line
…
last linE

sed -n ’s/e/E/gw out.txt’ le.txt

$ cat out.txt
linE onE Output
linE two is stored in
linE thrEE “out.txt”
… le
last linE

Sarfraaz Ahmed
fi
fi
fi
 Linux Course

sed Examples
• Delete blank lines : sed ‘/^$/ d’ file.txt

• Delete upto first blank line [ email header ]

• sed ‘1,/^$/ d’ file.txt

• Remove last character to convert Windows file to

Unix file format

• sed ’s/.$//‘ file.win > file.unix

Sarfraaz Ahmed

 Linux Course

Introduction to ‘awk’

Sarfraaz Ahmed
 Linux Course

awk
• Aho, Weinberger and Kernighan

• For manipulating structured data based on

patterns

• Data is interpreted in terms of

• rows / Records and nawk : New awk

gawk : GNU awk

• columns / Fields

• Generation of reports

Sarfraaz Ahmed

 Linux Course

awk format
Check if Perform actions
pattern is present for lines that match
in current line the pattern.

awk ‘ / pattern / { action } ’ le.txt

awk Read one
expects atleast Action line at a time
one pattern or statements should from input
one action be delimited by a le
semi colon

If no “pattern” is given, then perform actions for all lines

If no “action” is given, then print the lines that contain the pattern.

Sarfraaz Ahmed
fi
fi
 Linux Course

awk format
Fields NF

NR

{
Current records
ordinal
$1
john
$2
python
$3
ex01
$4
100
no. of elds
in current
record

{
number ben shell ex01 90
carl shell ex02 90 $0

ben shell ex02 95 prints entire

john python ex02 90 record
Records ben python ex01 80
ben shell ex03 100
carl shell ex03 80
john shell ex01 90
ben python ex02 75

data.txt

Sarfraaz Ahmed

fi

 Linux Course

awk format
• NR : Record Number
awk pattern { actions }
• NF : Number of Fields

• With $, gives field, otherwise number

• awk ‘{print NR, $(NF-1)}’ server.log

• 1 200 Second last

eld
• 2 200

Sarfraaz Ahmed
fi

 Linux Course

Printing Lines
$ awk ‘ { print } ’ data.txt Same
output
$ awk ‘ { print $0 } ’ data.txt
Print each
record john python ex01 100
ben shell ex01 90
carl shell ex02 90
ben shell ex02 95
john python ex02 90
ben python ex01 80
ben shell ex03 100
carl shell ex03 80
john shell ex01 90
ben python ex02 75

Sarfraaz Ahmed

 Linux Course

Printing Fields
$ awk ‘ { print $1, “\t: ”, $4 } ’ data.txt Same
output
$ awk ‘ { print $1, “\t: ”, $NF } ’ data.txt
Print Field john : 100
numbers 1 and 4 ben : 90
in each record carl : 90 john python ex01 100
ben shell ex01 90
ben : 95 carl shell ex02 90
ben shell ex02 95
john : 90 john python ex02 90
ben python ex01 80
ben : 80 ben shell ex03 100
carl shell ex03 80
ben : 100 john shell ex01 90
ben python ex02 75
carl : 80
john : 90
ben : 75

Sarfraaz Ahmed

 Linux Course

Print
Record number
Printing Fields
as well

$ awk ‘ { print NR, $1, “\t: ”, $4 } ’ data.txt Same

output
$ awk ‘ { print NR, $1, “\t: ”, $NF } ’ data.txt
Print Field 1 john : 100
numbers 1 and 4 2 ben : 90
in each record 3 carl : 90 john python ex01 100
ben shell ex01 90
4 ben : 95 carl shell ex02 90
ben shell ex02 95
5 john : 90 john python ex02 90
ben python ex01 80
6 ben : 80 ben shell ex03 100
carl shell ex03 80
7 ben : 100 john shell ex01 90
ben python ex02 75
8 carl : 80
9 john : 90
10 ben : 75

Sarfraaz Ahmed

 Linux Course

awk Script
$ awk ‘ { print NR, $1, “\t: ”, $NF } ’ data.txt
$ cat names.awk
{ Same
print NR " " $1 "\t: " $NF; output
}

$ awk -f names.awk data.txt

john python ex01 100
ben shell ex01 90
1 john : 100 carl
ben
shell
shell
ex02
ex02
90
95
2 ben : 90 john
ben
python
python
ex02
ex01
90
80
3 carl : 90 ben
carl
shell
shell
ex03
ex03
100
80
4 ben : 95 john
ben
shell
python
ex01
ex02
90
75
5 john : 90
6 ben : 80
7 ben : 100
8 carl : 80
9 john : 90
10 ben : 75

Sarfraaz Ahmed

 Linux Course

Print speci c lines

$ cat names.awk
/john/ {
print NR, " “, $1, "\t: “, $NF;
}
Print only
those lines that $ awk -f names.awk data.txt
contain john
john python ex01 100
ben shell ex01 90
carl shell ex02 90
1 john : 100 ben shell ex02 95
john python ex02 90
5 john : 90 ben python ex01 80
ben shell ex03 100
9 john : 90 carl shell ex03 80
john shell ex01 90
ben python ex02 75

Sarfraaz Ahmed

fi
 Linux Course

Comparing values
$ cat names.awk
$NF > 80 {
print NR, " “, $1, "\t: “, $NF;
}
Print only
those lines where $ awk -f names.awk data.txt
marks are > 80
1 john : 100 john
ben
python
shell
ex01
ex01
100
90

2 ben : 90 carl
ben
shell
shell
ex02
ex02
90
95

3 carl : 90 john
ben
python
python
ex02
ex01
90
80

4 ben : 95 ben
carl
shell
shell
ex03
ex03
100
80

5 john : 90 john
ben
shell
python
ex01
ex02
90
75

7 ben : 100
9 john : 90

Sarfraaz Ahmed

 Linux Course

Comparing values
$ cat names.awk
$2 ~ /py/ {
print NR, " “, $1, "\t: “, $NF;
~ [ tilde ] }
is used to
match $ awk -f names.awk data.txt
patterns in
john python ex01 100
elds ben shell ex01 90
carl shell ex02 90

1 john : 100 ben

john
shell
python
ex02
ex02
95
90

5 john : 90 ben
ben
python
shell
ex01
ex03
80
100

6 ben : 80 carl
john
shell
shell
ex03
ex01
80
90

10 ben : 75 ben python ex02 75

Sarfraaz Ahmed
fi

 Linux Course

BEGIN
BEGIN and END
$ cat names.awk
is executed BEGIN {
before any line count = 0; # initialisation
is read from } Executed
input for every line
$2 ~ /py/ { read from
count++; input
print NR, " “, $1, "\t: “, $NF;
}

END END { john python ex01 100

ben shell ex01 90
print "======================"; carl shell ex02 90
is executed ben shell ex02 95
print "Total entries : ", count; john python ex02 90
after all lines } ben python ex01 80
ben shell ex03 100
are read from carl shell ex03 80

input $ awk -f names.awk data.txt john

ben
shell
python
ex01
ex02
90
75

1 john : 100
5 john : 90
6 ben : 80
10 ben : 75
=================
Total entries : 4
Sarfraaz Ahmed

 Linux Course

Summation
Add up
$ cat names.awk marks from
/john/ {
each matched
total is total += $NF; line
initialised print NR, " “, $1, "\t: “, $NF;
to 0 by }
default END {
print "===================="; john python ex01 100
ben shell ex01 90
print "Total marks : ", total; carl shell ex02 90
ben shell ex02 95
} john python ex02 90
ben python ex01 80
ben shell ex03 100
carl shell ex03 80
$ awk -f names.awk data.txt john
ben
shell
python
ex01
ex02
90
75

1 john : 100
5 john : 90
9 john : 90
=================
Total marks : 280
Sarfraaz Ahmed

 Linux Course

Average
$ cat names.awk
/py/ {
pc++;
ptotal += $NF;
Multiple };
pattern and /sh/ {
action pairs sc++;
stotal += $NF;
are separated by john python ex01 100
} ben shell ex01 90
semicolon ; END { carl shell ex02 90
ben shell ex02 95
print "================================"; john
ben
python
python
ex02
ex01
90
80
print "Avg marks for Python: ", ptotal/pc; ben shell ex03 100
carl shell ex03 80
print "Avg marks for Shell : ", stotal/sc; john shell ex01 90
ben python ex02 75
}

$ awk -f names.awk data.txt

============================
Avg marks for Python: 86.25
Avg marks for Shell : 90.8333
Sarfraaz Ahmed

 Linux Course

Field Separator
john,python,ex01,100
ben,shell,ex01,90
FS Input Field Separator $ cat names.awk carl,shell,ex02,90
ben,shell,ex02,95
BEGIN { john,python,ex02,90
OFS Output Field Separator ben,python,ex01,80
FS=“," ; OFS=" : ";
ben,shell,ex03,100
RS="\n"; ORS=",\t";
carl,shell,ex03,80
} john,shell,ex01,90
RS Input Record Separator /john/ { ben,python,ex02,75
print NR,$1,$NF;
john python ex01 100
} ben shell ex01 90
ORS Output Record Separator carl shell ex02 90
ben shell ex02 95
john python ex02 90
ben python ex01 80
$ awk -f names.awk data.csv ben
carl
shell
shell
ex03
ex03
100
80
john shell ex01 90
ben python ex02 75

1 : john : 100, 5 : john : 90, 9 : john : 90,

OFS ORS

Sarfraaz Ahmed

 Linux Course

Conditional checks
$ cat names.awk
john python ex01 100
ben shell ex01 90
carl shell ex02 90
/ben/ { ben shell ex02 95
john python ex02 90
if ($2 == "shell") { ben python ex01 80
ben shell ex03 100
print $0; carl shell ex03 80
john shell ex01 90
total += $NF; ben python ex02 75

} else if ($2 == "python") {

py = py "\n" $0; ben shell ex01 90
ptotal += $NF;
String ben shell ex02 95
}; ben shell ex03 100
} Concatenation Total marks in shell : 285
END {
print "Total marks in shell :", total; ben python ex01 80
print py; ben python ex02 75
print "Total marks in Python :", ptotal; Total marks in Python : 155
}

$ awk -f total.awk data.txt

Sarfraaz Ahmed

 Linux Course

awk Examples

Extract date from the server.log file

evening.berkeley.edu - - [10/Jul/1995:22:18:17 -0400] "GET /shuttle/missions/missions.html HTTP/1.0" 200 8678

Date

awk ‘ { print $4 } ’ server.log

[10/Jul/1995:22:18:17

Sarfraaz Ahmed
 Linux Course

awk Examples

Extract date from the server.log file

evening.berkeley.edu - - [10/Jul/1995:22:18:17 -0400] "GET /shuttle/missions/missions.html HTTP/1.0" 200 8678

awk ‘ { print $4 } ’ server.log

[10/Jul/1995:22:18:17

awk ‘ { print $4 } ’ server.log | awk -F: ‘{print $1}’

[10/Jul/1995

Sarfraaz Ahmed
 Linux Course

awk Examples

Extract date from the server.log file

evening.berkeley.edu - - [10/Jul/1995:22:18:17 -0400] "GET /shuttle/missions/missions.html HTTP/1.0" 200 8678

awk ‘ { print $4 } ’ server.log

[10/Jul/1995:22:18:17

awk ‘ { print $4 } ’ server.log | awk -F: ‘{print $1}’

[10/Jul/1995

awk ‘ { print $4 } ’ server.log | awk -F: ‘{print $1}’ | sed “s/\[//”

10/Jul/1995
Sarfraaz Ahmed
 Linux Course

awk Examples

Find the total number of bytes transferred

evening.berkeley.edu - - [10/Jul/1995:22:18:17 -0400] "GET /shuttle/missions/missions.html HTTP/1.0" 200 8678

Bytes

awk ‘{ b += $(NF) } END {print b}’ server.log

Sarfraaz Ahmed
 Linux Course

awk Examples

Find total number of bytes transferred to berkeley.edu

evening.berkeley.edu - - [10/Jul/1995:22:18:17 -0400] "GET /shuttle/missions/missions.html HTTP/1.0" 200 8678

Bytes

awk $1 ~ /berkeley.edu/ ‘{ b += $(NF) } END {print b}’ server.log

Sarfraaz Ahmed
 Linux Course

awk Examples

john
Number only lines that have some text 1. john
ben
2. ben
3. carl
carl
4. ben
ben
5. john
john

awk 'NF {print $0}' names.txt | awk '{print NR " : " $0}'

Sarfraaz Ahmed

 Linux Course

awk Examples
Find the filename with the maximum size

ls -lSr | tail -1 | awk '{print $NF}'

wc -c * 2>/dev/null | head -n -1 | sort -n | tail -1 | awk '{print $NF}'

Get rid of
the “total”
line

Sarfraaz Ahmed
 Linux Course

awk Examples
Find the file with the maximum size

$5 > max {max=$5; maxline=$0}

END{print maxline}

ls -l | awk ‘$5 > max {max=$5; maxline=$0}; END{print maxline}’

$ ls -l | awk -f maxlines.awk
-rw-r--r-- 1 apple dev 544 Mar 7 2017 twelve.txt
$
But, fails to print all lines that have same file size !!!
Sarfraaz Ahmed

 Linux Course

Thats awk word

Find all the files with the maximum size
Hash
Map

$5 > max {max=$5; maxline[max]=maxline[max] ORS $0}

END{print maxline[max]}

$ ls -l | awk -f maxlines.awk
-rw-r--r-- 1 apple dev 544 Mar 7 2017 twelve.txt
-rw-r--r-- 1 apple dev 544 Oct 2 22:05 twelve.txt.bkp
$
Sarfraaz Ahmed

 Linux Course

THANK YOU

Sarfraaz Ahmed