Training

Linux Command Everyone Should Know

What Linux is (and what it isn’t)

Background
 Linux is a kernel
 Foundation of the GNU/Linux OS
 User commands are from GNU utilities
o Bash (default shell on most distributions) has many commands built in.
o Utilities like sed, awk, grep, etc.
o Everything that lies outside the kernel.
 Most of what people call “linux” is actually a collection of GNU utilities and the Linux kernel assembled into a
distribution.
o ForeScout uses a customized version of CentOS, which in turn is Red Hat Enterprise, but with the Red Hat
copyrighted pieces (logos and other artwork) replaced.
 You can download packages from the mirror links @ https://www.centos.org/download/mirrors/.

So what can you do with it?

 Whatever you want
 Extremely powerful and flexible
 Support though email lists, websites, and commercial support providers (Red Hat, etc.)
 Source code is widely available to see what makes it “tick” or modify to suit your needs/wants.

How does it work?

 The kernel handles IO tasks, memory management, and all other system-level functionality.
 The utilities sit on top of that to make a complete OS.
 Commands can be chained, or piped, to give greater functionality.
 You can also create shell scripts to automate tasks.
o Here’s an example of a small shell script.
 #! /bin/bash
 list=`ls *.tgz`
 for f in $list
 do
 tar –zxf $f

1
 Training

 done
o Breaking down the script
 The first line specifies the shell to run the script.
 This way you can specify a different shell than the one you’re in.
 Examples in /bin/cshell, /usr/bin/perl, and many others.
 The second line creates a variable “list” using the command “ls *.tgz”.
 The third line creates another variable “f”, and says for every entry “f” in the list “list”…
 Do the following command(s)
 tar –zxf ($f the file from the list)
 And done, there are no more commands.
 Scripts can be as simple or complex as desired.
 You can write whole programs in shell script.
 For common tasks we do here in support, putting together a collection of scripts do run through common data
collections or to set up debug environments is recommended.
 Maybe most interesting; everything is a file.
o Any command that works against a file will work against anything, since it’s all files.

Everything you never wanted to know about the Linux CLI, or 1001* super handy dandy really great
Linux shell commands with examples (*actual number may vary)

 ls
o List file command, with a lot of optional switches
o ls –lh – list files in long form (-l) and human readable output (-h)

$ ls -lh
total 21M
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Desktop
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Documents
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Downloads
-rwxr-xr-x 1 rhughes Domain Users 21M Oct 17 14:15 ForeScout-va-10.2.5.7-102057001_unsigned.fpi
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Music
lrwxrwxrwx 1 rhughes Domain Users 38 Jul 18 13:02 mydocs -> '/cygdrive/c/Users/rhughes/My
Documents'
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Pictures
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Public
-rwxr-xr-x 1 rhughes Domain Users 582K Oct 17 14:16 rxvt-2.7.10.tar.gz
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Templates
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Videos
drwxr-xr-x+ 1 rhughes Domain Users 0 Oct 7 17:29 work

2
 Training

o ls –S will sort by size with the largest first

$ ls -hlS
total 21M
-rwxr-xr-x 1 rhughes Domain Users 21M Oct 17 14:15 ForeScout-va-10.2.5.7-102057001_unsigned.fpi
-rwxr-xr-x 1 rhughes Domain Users 582K Oct 17 14:16 rxvt-2.7.10.tar.gz
lrwxrwxrwx 1 rhughes Domain Users 38 Jul 18 13:02 mydocs -> '/cygdrive/c/Users/rhughes/My
Documents'
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Desktop
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Documents
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Downloads
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Music
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Pictures
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Public
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Templates
drwxr-xr-x+ 1 rhughes Domain Users 0 Apr 28 10:30 Videos
drwxr-xr-x+ 1 rhughes Domain Users 0 Oct 7 17:29 work

$ ls -a
. .profile Music
.. .ssh mydocs
.bash_history .viminfo Pictures
.bash_profile .Xauthority Public
.bashrc .xsession-errors rxvt-2.7.10.tar.gz
.config Desktop Templates
.dbus Documents Videos
.inputrc Downloads work
.minttyrc ForeScout-va-10.2.5.7-102057001_unsigned.fpi

o ls –a will show all files and directories, including hidden files and directories
o ls -t

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
3
 Training

.:
Desktop
Documents
Downloads
ForeScout-va-10.2.5.7-102057001_unsigned.fpi
meh.txt
Music
mydocs
Pictures
Public
rxvt-2.7.10.tar.gz
Templates
Videos
work

./Desktop:
…
./work/rxvt-2.7.10/W11/X11:
cursorfont.h
Intrinsic.h
keysym.h
keysymdef.h
Xatom.h
Xlib.h
Xlocale.h
Xmd.h
Xutil.h

o ls –R will show recursive file listing within a directory.

 head (stop laughing, you immature jerk)
o Displays lines from the beginning of a file.
o Displays the first 10 lines if no argument except the file name is given.

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
4
 Training

o Most common argument specifies number of lines to display.

 Head -50 will display the first 50 lines of a file
 tail (I said stop laughing!)
o Does the opposite of head; displays the last lines of a file.
o Displays the last 10 lines by default.
 tail –50 will display the last 50 lines of a file.
o Can be used to watch as text is added to a file.
 tail –f <filename>
 Very useful for watching a log file.
 Can also be piped to grep to only display specific data.
 less
o A more advanced version of more
o Supports searches via the “/” key
o Search can also be combined with regular expressions.
o There are lots of other options, but there are some of the most common:
 m Followed by any lowercase letter, marks the current position with that letter.
 ' (Single quote.) Followed by any lowercase letter, returns to the position which was previously
marked with that letter. Followed by another single quote, returns to the position at which the last
"large" movement command was executed. Followed by a ^ or $, jumps to the beginning or end of
the file respectively. Marks are preserved when a new file is examined, so the ' command can be
used to switch between input files.
 ^X^X Same as single quote.
 /pattern Search forward in the file for the N-th line containing the pattern. N defaults to 1. The pattern
is a regular expression, as recognized by the regular expression library supplied by your system.
The search starts at the second line displayed (but see the -a and -j options, which change this).
 Certain characters are special if entered at the beginning of the pattern; they modify the type of
search rather than become part of the pattern:
 ^N or ! Search for lines which do NOT match the pattern.
 ^E or * Search multiple files. That is, if the search reaches the END of the current file without finding
a match, the search continues in the next file in the command line list.
 ^F or @ Begin the search at the first line of the FIRST file in the command line list, regardless of
what is currently displayed on the screen or the settings of the -a or -j options.
 ^K Highlight any text which matches the pattern on the current screen, but don't move to the first
match (KEEP current position).
 ^R Don't interpret regular expression metacharacters; that is, do a simple textual comparison.

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
5
 Training

 ? pattern Search backward in the file for the N-th line containing the pattern. The search starts at the
line immediately before the top line displayed.
 grep – the bacon of shell commands
o Stands for Global Regular Expression Print
 You really wanted to know that, didn’t you?
o Variations include egrep and fgrep
 grep –e and egrep are the same thing.
 fgrep and grep –F are the same thing.
o The grep command is most powerful when combined with regular expressions.
 Covered lightly now, more in-depth coverage for regular expressions later.
o The basic usage is simply “grep < expression > file”.

$ grep make Makefile

# This variable makes it possible to move the installation root to another
# You can run eg. 'make install DESTDIR=/packages/rxvt-xx' to accomplish
@for I in $(allsubdirs); do (cd $$I; $(MAKE) $@ || (echo "Failed to make distclean in $$I"; exit 0) );
done

o Or find out which files in a directory contain the search term with “grep –l < expression > [dir | file | wildcard
files]”.

$ grep -l make *
grep: autoconf: Is a directory
ChangeLog
config.log
configure
grep: doc: Is a directory
INSTALL
libtool
Makefile
Makefile.in
grep: rclock: Is a directory
README.configure
rxvt-2.7.10.lsm
grep: src: Is a directory
grep: W11: Is a directory

o You can also use grep to find out the line number your search term is in with “grep –n <expression> [dir | file |
wildcard files]”.

$ grep -n auto Makefile

4:# autoconf/Make.common.in -*- Makefile -*-
91:DIST_CFG = autoconf/aclocal.m4 autoconf/xpm.m4 autoconf/libtool.m4 \
92:
© 2016.autoconf/configure.in autoconf/config.h.in
ForeScout Technologies, Inc. \
is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
93: autoconf/Make.common.in
CounterACT Edge, ActiveResponse andautoconf/install-sh autoconf/mkinstalldirs
CounterACT are trademarks \
or registered trademarks of ForeScout. Other names
94: autoconf/config.guess autoconf/config.sub \
mentioned may be trademarks of their respective owners.
95: autoconf/ltmain.sh \
97:MKDIR = $(srcdir)/autoconf/mkinstalldirs 6
110: $(RMF) -r autom4te.cache
130:configure: autoconf/configure.in autoconf/aclocal.m4 autoconf/config.h.in
138:autoconf/config.h.in: autoconf/configure.in
163: (cd $(srcdir); $(RMF) Makefile autoconf/Make.common)
167: mkdir ../$(VERNAME)/autoconf;
 Training

o Highlight matches as in the last example by using the following: grep --color, egrep --color=auto, or fgrep --
color=auto in addition to the other arguments.
o Find everything that doesn’t match the expression by using grep –v <expression>.
o Find everything that matches the expression regardless of case with “grep –i <expression>”.
o Match an expression plus X lines after the match by using “grep –A <NUM> <expression>”
o Match an expression plus X lines before the match by using “grep –B <NUM> <expression>”
o Include subdirectories in the search with “grep –r <expression> <wildcard mask>”.
 The wc (word count) command is useful for finding out how many lines are in a file or in the output of another
command.
o Run “wc –l” to get a line count.
 The sort command is also very useful.
o sort –u Sort unique
o sort –n Sort numerically
 The “find” command is useful for finding files.
 The find command has a number of options, so check the man page for complete details.
o Some of the most common are “-type”, “-size”, “-user”, “-group”, “-name”, and many, many others.
o These are most useful when used in the same command, such as “find –type f –user root –name “*filename*”.
 You can optionally run a command against the found file.
o For example, if you want to find all regular files and run a grep against them, run “find –type f –exec grep
<something> {} \;”
 The “-exec” switch means exec the command that follows.
 The “{}” characters stand for the file and path the find command located.
 The “\;” terminates the command.
 Get and/or manipulate system information with the following commands:
o netstat – very versatile, and can show everything from routes and connections to interface statistics. Add “-c”
to any of the switches to get real-time monitoring similar to top.
 Use netstat –rn to show routing
 There are several flags that will be shown in the Flags column that help to understand the
routes:
 A Receive all multicast at this interface.
 B OK broadcast.
 D Debugging ON.
 M Promiscuous Mode.
 O No ARP at this interface.

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
7
 Training

 P P2P connection at this interface.

 R Interface is running.
 U Interface is up.
 G Not a direct entry.
 The route showing either the word “default” or 0.0.0.0 will be the default route.
 Routes are ordered from most specific to least specific. Thus, the default gateway will always show
last, while host routes will always show first.
o You can add and delete routes from the command line like this:
 If you want to add a route, the syntax is “route add –net [<network>/<CIDR mask> |<network>
netmask <dotted quad netmask> gw <gateway IP>”
 Note that you can add a route to the same network but with different gateways, including
the default gateway, which can be confusing. The order in which the routes appear is the
order in which they’re processed.
 To delete a route, the syntax is “route del –net [<network>/<CIDR mask> |<network> netmask
<dotted quad netmask>.
 If you accidently got routes for the same network in multiple times but with different
gateways, only one of which is correct, then if you also specify “gw <gw IP>” you can
delete only that specific route. Otherwise, the first route that matches the network will be
deleted.
 Use netstat –i to get interface stats.

Kernel Interface table

Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 22046279 0 0 0 6951285 0 0 0 BMRU
lo 65536 0 10578011 0 0 0 10578011 0 0 0 LRU

 Output is:
o Iface – interface name
o MTU – MTU size
o RX-OK – receive OK
o RX-ERR – receive errors
o RX-DRP – receive drops
o RX-OVR – receive overruns
o TX-OK – transmit ok
o TX-ERR – transmit errors

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
8
 Training

o TX-DRP – transmit drops

o TX-OVR – transmit overruns
o Flg – A = receive multicast; B = broadcast allowed; D = debugging turned on; L =
loopback interface; M = all packets received; N = trailers avoided; O = no arp on
this interface; P = point-to-point interface; R = interface is running; U = interface
is up.
 Use netstat –a or –na to see all listening/connected ports and sockets. Use netstat –nap to
also see the pid of the process listening.
o State is an important column and tells you whether the port is making the
connection, connected, or ending the connection.
 If you see syn_sent, it means that the initial syn packet has been sent,
but the connection has not been established yet.
 When a connection stays in this state for more than (typically) a second
or two, it tends to indicate that the connect cannot be established
because either the port is blocked or nothing is listening at the other
end.
 When you see close_wait, the server has received a FIN packet. Large
numbers of connections in close_wait can indicate a file descriptor leak
or buggy app/plugin.
 Another way to look at routing and interface information is with the ip command.
 Routes are shown with ip route show
o Also shows dynamic routes, not just kernel routes as with netstat –r
 Other useful commands are ip link show and ip addr show.
o These show the link state and IP addressing information, respectively.
 Use ifconfig to see interface information
 Ifconfig –a will show all interfaces, with or without an IP address.
 Also contains basic traffic information like netstat –i

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
9
 Training

[root@RHughes-CA1 ~]# ifconfig -a

eth0 Link encap:Ethernet HWaddr 00:50:56:95:6F:1A
inet addr:10.100.11.42 Bcast:10.100.11.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe95:6f1a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:22300331 errors:0 dropped:0 overruns:0 frame:0
TX packets:3726872 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1929686090 (1.7 GiB) TX bytes:1214849868 (1.1 GiB)

eth1 Link encap:Ethernet HWaddr 00:50:56:95:46:A2

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:915608590 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:185170481235 (172.4 GiB) TX bytes:0 (0.0 b)

eth2 Link encap:Ethernet HWaddr 00:50:56:95:66:27

UP BROADCAST RUNNING MULTICAST MTU:1505 Metric:1
RX packets:23000706 errors:0 dropped:0 overruns:0 frame:0
TX packets:2498798 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1652214017 (1.5 GiB) TX bytes:117093365 (111.6 MiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:15793287 errors:0 dropped:0 overruns:0 frame:0
TX packets:15793287 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2884938348 (2.6 GiB) TX bytes:2884938348 (2.6 GiB)

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
10
 Training

 Get disk usage data via the “df” command.

 Running “df –h” outputs the data in human readable format.

[root@RHughes-CO6-64 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_centos6x-lv_root
12G 4.0G 7.0G 37% /
tmpfs 939M 0 939M 0% /dev/shm
/dev/sda1 485M 60M 401M 13% /bootv

o As you can see, long mount point names sometimes cause odd wrapping.
o This is a limitation of the utility.
 Learn what disk slice is mounted where with the “mount” command.

[root@RHughes-CO6-64 ~]# mount

/dev/mapper/vg_centos6x-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

 Get information on directory and file sizes by using du.

 Running du with no arguments results in sizes for the current directory, files, and
subdirectories.
 Running du –s gives a summary for all files and subdirectories within the current directory.
 Running du –h will give the output in human readable format.
 Running free will give you useful memory stats

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
11
 Training

[root@RHughes-CO6-64 ~]# free

total used free shared buffers cached
Mem: 1922628 936940 985688 0 124560 398328
-/+ buffers/cache: 414052 1508576
Swap: 4128760 0 4128760

 Stats are in kilobytes.

 Running free –l will give you stats on low and high memory separately.
o This isn’t really relevant on machines with OSUP installed, as they run 64 bit
kernels which can directly map terabytes of memory into the “low” space.
 Running “free –s <number>” will update every <number> seconds.
 Another useful memory/processor stats util is vmstat
 Running “vmstat” alone will display a summary of a number of statistics about the system.

[root@RHughes-CO6-64 ~]# vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 982572 126004 398368 0 0 2 8 137 14 3 4 94 0 0

 Here’s a list, in order, of what each column means:

o Running
o Sleeping
o Total swap used, in kilobytes
o Total free memory
o Total buffer memory
o Total disk cache memory usage
o Memory swapped in from disk, in KB/sec
o Memory swapped out to disk, in KB/sec
o Blocks read in from I/O devices (blocks/sec)
o Blocks written to I/O devices (blocks/sec)
o Interrupts/sec
o Context switches/sec

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
12
 Training

o Userland CPU usage in %

o Kernelspace CPU usage in %
o CPU idle %
o CPU IO wait %
 You can use the “file” command to find out what type of file a file is.
 Very useful for core files, as it will tell you both the exact process a core came from as well
as whether the core is useful.

[root@RHughes-CO6-64 Documents]# file SecureConnectorLinuxInstall.sh

SecureConnectorLinuxInstall.sh: POSIX shell script text executable
[root@RHughes-EM1 mick crash]# file core.48981
core.48981: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'trails -o log/engine.log -e log/engine.err -i DFLT_IF_CFG
-U -A -S 5 -W 50000 -'
[root@RHughes-EM1 mick crash]# file UOG-cores-2_annu3.tgz
UOG-cores-2_annu3.tgz: gzip compressed data, from Unix, last modified: Wed Sep 28 08:10:35 2016

o If you run “file <core>” and the result is “data”, then the core is just garbage
dumped from memory and not a dump of the running process, so it’s not useful
for investigation.
 Using the kill command
 Very straightforward, just run “kill <pid>” to kill a process with the normal termination.
 There are a number of other kill options as well when running “kill –SIGNAL_NUMBER
<pid>”
 The most common is “kill -9 <pid>”, which is the SIGKILL (brutal kill) signal.
 Another useful one is “kill -1 <pid>”, which is SIGHUP. This tells the process to “hangup”,
or re-read it’s config files but continue running.
 Running “kill -6 <pid>” will force the process to dump core.
 The “pidof” command will return the pid(s) of the named process process.

[root@RHughes-CO6-64 ~]# pidof sshd

22512 21461 2773

 You can run “strace” to see system calls.

 The output from this command requires a deep knowledge of Linux system calls.
 Run “strace –f –s 1024 –o <output file> <command>”.

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
13
 Training

 This will dump the output to a file and follow processed forked from the main process.
 You can also attach to a running process with “strace –f –s 1024 –o <output file> -p <pid of
process>”.
o You can view network traffic with the “tcpdump” command.
 Run “tcpdump –s 0 –i <interface> -w <file name>”
 This captures the entire packet (default is the first 32 bytes, which is just the headers and a little
data) on the interface specified by –i and writes to the file <file name>.
 The files can be opened with wireshark.
 If you omit the –w, then basic output is written to the screen.
 You can also filter on hosts or ports.
 Run “tcpdump –i <interface> host host1 and port port1”
 This is also very flexible and you can chain as much as you want together.
 Look up a tutorial on Berkley Packet Filter language for in-depth details.
 Brief words about sed and awk
o Unlike other utilities, awk supports its own scripting.
o So many things can be done with awk that exploring them in any depth is beyond the scope of this document,
but a session specifically on awk might be done.
o A few examples of very basic usage will be presented later.
o The sed command is for stream editing.
 The name “sed” actually stands for “stream editor”.
 The most common use is when you need to change the same word in multiple places in one or more
files.
 The syntax is “sed –i ‘s/old_word/new_word/g’ file”
 The “–i” switch mean “in place”.
 The “g” mean global, or all occurrences. Otherwise, only the first occurrence sed finds will
be changed.
 I/O redirection in BASH
o Our appliances use the BASH shell (bourne again shell) as the default.
o You will mainly be concerned with standard out (stdout), which is the normal output from a command, and
standard error (stderr) which are things marked as errors.
 Sometimes for some unknowable reason developers will choose to send all output to stderr.
 They should die. Or at least have their keyboards taken away.
o Occasionally standard in (stdin) will be useful when you’re scripting.
o Redirection is straight forward.

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
14
 Training

 Remember, everything in Linux is a file, with an associated file descriptor.

 Since everything is a file with a descriptor, just remember that stdout is “1” and stderr is “2”.
 Output from commands is sent either to stdout or stderr, or sometimes some messages are sent to
one while other messages are sent to the other.
 If you want to redirect stdout, then use “> some_file” to capture the output from the command.
 If you only want things flagged as errors, use “2 > some_file”.
 Be aware the some developers have marked their entire output to go to stderr, so if you’re
trying to redirect stdout but still getting screens full of data, see if all the output is going to
stderr.
 If you want both, then the syntax is “my_command > some_file 2>&1”.
 This tells the shell to send stderr to the same place that stdout is sent.
 You can also redirect either or both through a “pipe”, which is the “|” symbol.
 This is most useful when you want to parse the output of a command through another command.
 You can also chain multiple unlimited commands by piping the output of one into another.
 The best example is something like “grep <short_word> <file> |grep –v
<short_word_other_characters>”
 This allows you to match on only the work you want, even if there are other similar words
in the data.

But first, a bit about regular expressions (regex)

 There are many kinds of regex; perl, shell, java, etc.
 A full tutorial on any of these is beyond the scope of this document, but this will be enough to get you started.
o Google “regular expression tutorial” for complete lessons, examples, and regex tests.
 You’re probably familiar already with a bit of regex.
 When you do something like dir file* or ls file* that’s a wildcard, which is also much like a regex.
 Let’s start with the basic definition of terms
o Literal - A literal is any character we use in a search or matching expression, for example, to find ind in
windows the ind is a literal string - each character plays a part in the search, it is literally the string we want to
find.
o Metacharacter - A metacharacter is one or more special characters that have a unique meaning and are NOT
used as literals in the search expression, for example, the character ^ (circumflex or caret) is a metacharacter.
o Target string - This term describes the string that we will be searching, that is, the string in which we want to
find our match or search pattern.
o Search expression - Most commonly called the regular expression. This term describes the search expression
that we will be using to search our target string, that is, the pattern we use to find what we want.

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
15
 Training

o Escape sequence - An escape sequence is a way of indicating that we want to use one of our metacharacters
as a literal. In a regular expression an escape sequence involves placing the metacharacter \ (backslash) in
front of the metacharacter that we want to use as a literal, for example, if we want to find (s) in the target string
window(s) then we use the search expression \(s\) and if we want to find \\file in the target string c:\\file then
we would need to use the search expression \\\\file (each \ we want to search for as a literal (there are 2) is
preceded by an escape sequence \).
 This is what the metacharacters do
o Brackets, Ranges and Negation
 [ ] Match anything inside the square brackets for ONE character position, once and only once. For
example, [12] means match the target to 1 and if that does not match then match the target to 2
while [0123456789] means match to any character in the range 0 to 9.
 - The - (dash) inside square brackets is the “range separator” and allows a range to be defined. In
our example above of [0123456789] we could rewrite it as [0-9].
 You can define more than one range inside a list, for example, [0-9A-C] means check for 0
to 9 and A to C (but not a to c).
 NOTE: To test for - inside brackets (as a literal) it must come first or last, that is, [-0-9] will
test for - and 0 to 9.
 ^ The ^ (circumflex or caret) inside square brackets negates the expression. We will see an alternate
use for the circumflex/caret outside square brackets later. Example: [^Ff] means anything except
upper or lower case F and [^a-z] means everything except lower case a to z.
 Notes: There are no spaces between the range delimiter values, if there was, depending
on the range, it would be added to the possible range or rejected as invalid. Be very careful
with spaces.
 Some regular expression systems, notably VBScript, provide a negation operator (!) for
use with strings. This is a non-standard feature and therefore the resulting expressions are
not portable.
 Because of the dual nature of the caret you will frequency see expressions like [^"], [^<] or
[^,] which are typically used as separator triggers (when combined with iterations) for more
complex searches or when parsing HTML, comma delimited text, etc.
o Positioning (or Anchors)
 ^ The ^ (circumflex or caret) when not used inside square brackets (where it has a diffent meaning)
means look only at the beginning of the target string. For example, ^Win will not find Microsoft
Windows but ^Moz will find Mozilla Firefox.
 $ The $ (dollar) means look only at the end of the target string, for example, fox$ will find a match in
'silver fox' since it appears at the end of the string but not in 'the fox jumped over the moon'.
 . The . (period) means any character(s) in this position, for example, ton. will find tons, tone and tonn
in tonneau but not wanton because it has no following character. Further, ton. would find “wanton “
because of the trailing space.

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
16
 Training

o Iteration 'metacharacters'
 ? The ? (question mark) matches when the preceding character occurs 0 or 1 times only, for
example, colou?r will find both color (u is found 0 times) and colour (u is found 1 time).
 * The * (asterisk or star) matches when the preceding character occurs 0 or more times, for
example, tre* will find tree (e is found 2 times) and tread (e is found 1 time) and trough (e is found 0
times and thus returns a match only on the tr).
 + The + (plus) matches when the preceding character occurs 1 or more times, for example, tre+ will
find tree (e is found 2 times) and tread (e is found 1 time) but NOT trough (0 times).
 {n} Matches when the preceding character, or character range, occurs n times exactly, for example,
to find a local phone number we could use [0-9]{3}-[0-9]{4} which would find any number of the form
123-4567. Value is enclosed in braces (curly brackets).
 Note: The - (dash) in this case, because it is outside the square brackets, is a literal.
Louise Rains writes to say that it is invalid to commence a NXX code (the 123) with a zero
(which would be permitted in the expression above). In this case the expression [1-9][0-9]
{2}-[0-9]{4} would be necessary to find a valid local phone number.
 {n,m} Matches when the preceding character occurs at least n times but not more than m times, for
example, ba{2,3}b will find baab and baaab but NOT bab or baaaab. Values are enclosed in braces
(curly brackets).
 {n,} Matches when the preceding character occurs at least n times, for example, ba{2,}b will find
'baab', 'baaab' or 'baaaab' but NOT 'bab'. Values are enclosed in braces (curly brackets).
o More 'metacharacters'
 () The ( (open parenthesis) and ) (close parenthesis) may be used to group (or bind) parts of a
search expression together. Officially this is called a subexpression (a submatch or group) and
subexpressions may be nested to any depth. Parentheses (subexpresions) also capture the
matched element into a variable that may be used as a backreference
 | The | (vertical bar or pipe) is called alternation in techspeak and means find the left hand OR right
values, for example, gr(a|e)y will find 'gray' or 'grey'. Having found the literal characters 'gr' - if the
first test is not valid (a) the second will be tried (e), if the first is valid the second will not be tried.
Alternation can be nested within each expression, thus gr((a|e)|i)y will find 'gray', 'grey' and 'griy'.

Putting it all together into something that will help you

 Let’s look at some examples of putting all this lovely newfound knowledge together.
o Finding all lines with an IP addresses in a file:
 egrep "([0-9]{1,3}[\.]){3}[0-9]{1,3}" <file>
o Find a line in a debug output from va containing a registry key and the 6 lines after it:
 grep –A6 SOFTWARE\\\\Network\ Associates\\\\TVD\\\\VirusScan\ Enterprise\\\\CurrentVersion
 The interesting thing about this expression is that each part of the registry key contains two back
slashes in the name. But in order to find a literal backslash, you have to escape the character, or put
an escape character (the backslash, ironically enough) in front of the character you want to search

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
17
 Training

as a literal (remember that a literal in this context is the actual character without its special meaning).
In this case, you’re escaping two backslashes in order to find a literal “\\”. Normally this expression
would find only a single “\”. We also have a space, so we need to escape that as well. Otherwise,
the grep command would see the part of the expression before the space as the whole expression
and the part after as the file to grep.
o Or another way would be:
 egrep –A6 ^subkey
 The word “subkey” prefaces registry keys, so this matches only lines that start with this
specific word.
 find And remember the ^ metacharacter means that the expression only matches lines that
start with the expression, whereas “word$” only matches lines ending with the expression.
o Find all occurrences of the word “error”, then filter it down to those occurrences which also contain a specific
IP address:
 grep error *.log|grep 12.4.5.87
o Find all occurrences of the work “error”, filter that down to those occurrences containing a specific IP, then
further filter the output to that which doesn’t contain the phrase “fingerprint=Windows Machine”
 grep error *.log |grep 172.27.43.177 |grep -v fingerprint\=Windows\ Machine
o Tail a logfile while looking for a specific IP or phrase.
 tail –f </path/to/file> |grep <ip address>|<phrase>
o Find all instances of a MAC address in a log.
egrep "([0-9a-fA-F][0-9a-fA-F]:){5}([0-9a-fA-F][0-9a-fA-F])" <logfile>
o A couple of tcump dump examples:
 tcpdump –i eth0 –s0 host 1.2.3.4 and host 4.3.2.1 port 80
 This captures all traffic using a source or destination port of 80 between the hosts 1.2.3.4
and 4.3.2.1.
 tcpdump –i eth0 not port 22
 Captures everything that is not using a source or destination port of 22.
 tcpdump –i eth0 src host 1.2.3.4 and dst host 4.3.2.1
 Captures traffic leaving 1.2.3.4 for 4.3.2.1
 tcpdump –i eth0 ether host <mac address>
 Captures traffic to or from a specific MAC address regardless of IP.
 tcpdump –i eth0.100
 Captures traffic on eth0 vlan 100.
 tcpdump –i eth0 arp
 Captures all arp traffic on eth0.

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
18
 Training

 And all the rest of the syntax can be found at http://biot.com/capstats/bpf.html

o

© 2016. ForeScout Technologies, Inc. is a privately held Delaware corporation. ForeScout, the ForeScout logo, ControlFabric,
CounterACT Edge, ActiveResponse and CounterACT are trademarks or registered trademarks of ForeScout. Other names
mentioned may be trademarks of their respective owners.
19