Copyright © 1991 Bruce Barnett and General Electric Company

Copyright © 2001, 2011, 2013 Bruce Barnett All Rights reserved

Original version written in 1991 and published in the Sun Observer

Thanks to Jesse Silverman for corrections, and @mathias

Updated Thu Aug 11 21:37:24 EDT 2011

Before you study regular expressions, it is important that you understand how to use
the shell to send regular expressions to a program like grep and sed.

Regular expressions use meta-characters. The shells also have meta-characters. Meta-
characters are simply characters that have a special meaning. The problem occurs
when you want to use a regular expression in a shell script. Will the shell do something
special with the character? Or will it be passed unchanged to the program?

If you want to send a meta-character to a program, you must quote the meta-
character - that is - you must tell the shell to leave it alone.

The "$" character is a good example. It could be the beginning of a variable name, or it
could be part of a regular expression. If you need a regular expression, you must know
if any of the characters of the expression are meta-characters, and must know the right
way to quote that character, so that it is passed to the program without being modified
by the shell.

I wrote this tutorial in a shell-agnostic manner. That is, what I describe here holds for
all of the major varieties of shells. In other words, it doesn't matter which shell you
use - because they all understand the three methods of quoting/escaping
meta-characters - more or less. There are some tricky bits, but I will mention them
later when I cover the different shells.

Here is a chart of the meta-characters the Bourne and C shell know about. I have also
included several combinations of characters just to make this table more complete.
There is a lot of detail on this chart.

List of Special Characters and what they

mean

Character Where Meaning

<RETURN> csh, sh Execute command
# csh, sh, ASCII files Start a comment
<SPACE> csh, sh Argument separator
` csh, sh Command substitution
" csh, sh Weak Quotes
' csh, sh Strong Quotes
\ csh, sh Single Character Quote
variable sh, csh Variable
variable csh, sh Same as variable
| csh, sh Pipe character
^ sh Pipe Character
& csh, sh Run program in background
? csh, sh Match one character
* csh, sh Match any number of characters
; csh, sh Command separator
;; sh End of Case statement
~ csh Home Directory
~user csh User's Home Directory
! csh History of Commands
- Programs Start of optional argument
$# csh, sh Number of arguments to script
$* csh, sh Arguments to script
$@ sh Original arguments to script
$- sh Flags passed to shell
$? sh Status of previous command
$$ sh Process identification number
$! sh PID of last background job
&& sh Short-circuit AND
|| sh Short-circuit OR
. csh, sh Typ. filename extension
. sh Source a file and execute as command
: sh Nothing command
: sh Separates Values in environment variables
: csh Variable modifier
Character Where Meaning
[] csh, sh Match range of characters
[] sh Test
%job csh Identifies job Number
(cmd;cmd) csh. sh Runs cmd;cmd as a sub-shell
{} csh In-line expansions
{cmd;cmd } sh Like (cmd;cmd ) without a subshell
>ofile csh, sh Standard output
>>ofile csh, sh Append to standard output
<ifile csh, sh Standard Input
<<word csh, sh Read until word, substitute variables
<<\word csh, sh Read until word, no substitution
<<-word sh Read until word, ignoring TABS
>>!file csh Append to file, ignore error if not there
>!file csh Output to new file, ignore error if not there
>&file csh Send standard & error output to file
<&digit sh Switch Standard Input to file
<&- sh Close Standard Input
>&digit sh Switch Standard Output to file
>&- sh Close Standard Output
digit1<&digit2 sh Connect digit2 to digit1
digit<&- sh Close file digit
digit2>&digit1 sh Connect digit2 to digit1
digit>&- sh Close file digit

I am not going to cover each one of these special meta-characters.

What is important is a solid understanding of the characters that have these special
meanings. I will also discuss how you can verify the shell is interpreting the special
characters, so you can pinpoint where your problem lies.

There are three different "quotation" marks on the keyboard.. Two of them use marks
used for quotations in English usage, and are sometimes called the single quote and
double quote.. The third quotation mark is the back quote (more properly called
backtick or grave) character: "`". It looks like the single quote and some times people
get them confused in shell scripts. The first two are used for quoting phrases in Unix.
The back quote is not used for quoting characters. That character is used for command
substitution, where the characters between them are executed by the shell and the
results is inserted on that line. Example:

% echo the date is `date`

Let's get back to the Unix quoting mechanisms. The three quoting mechanisms you can
use are the single quote, and double quote, and the backslash.

Quoting a single character

with the backslash

You can prevent the shell from interpreting a character by placing a backslash ("\") in
front of it. Here is a shell script that can delete any files that contain an asterisk:

echo This script removes all files that

echo contain an asterisk in the name.
echo
echo Are you sure you want to remove these files\?
rm -i *\**

The backslash was also necessary before the question mark, which is also a shell meta-
character. Without it, the shell would look for all files that match the pattern "files?." If
you had the files "files1" and "files2" the script would print out
Are you sure you want to remove these files1 files2

which is not what you want.

The backslash is the "strongest" method of quotation. It works when every other
method fails. If you want to place text on two or more lines for readability, but the
program expects one line, you need a line continuation character. Just use the
backslash as the last character on the line:

% echo This could be \

a very \
long line\!
This could be a very long line!
%

This escapes or quotes the end of line character, so it no longer has a special meaning.
In the above example, I also put a backslash before the exclamation point. This is
necessary if you are using the C shell, which treats the "!" as a special character. If you
are using some other shell, it might not be necessary.

Strong Quoting with the

Single Quotes

When you need to quote several character at once, you could use several backslashes:

% echo a\ \ \ \ \ \ \ b

(There are 7 spaces between 'a' and 'b'.) This is ugly but works. It is easier to use pairs
of quotation marks to indicate the start and end of the characters to be quoted:

% echo 'a b'

(The HTML ruins the formatting. Imagine that there are 7 spaces between the a and b.
-Bruce) Inside the single quotes, you can include almost all meta-characters:

% echo 'What the *heck* is a $ doing here???'

What the *heck* is a $ doing here???

The above example uses asterisks, dollar signs, and question marks meta-characters.
The single quotes should be used when you want the text left alone. If you are using
the C shell, the "!" character may need a backslash before it. It depends on the
characters next to it. If it is surrounded by spaces, you don't need to use a backslash.
 Weak Quotes with the
Double Quotes

Sometimes you want a weaker type of quoting: one that doesn't expand meta-
characters like "*" or "?," but does expand variables and does command substitution.
This can be done with the double quote characters:

% echo "Is your home directory $HOME?"

Is your home directory /home/kreskin/u0/barnett?
% echo "Your current directory is `pwd`"
Your current directory is /home/kreskin/u0/barnett

Once you learn the difference between single quotes and double quotes, you will have
mastered a very useful skill. It's not hard. The single quotes are stronger than the
double quotes. Got it? Okay. And the backslash is the strongest of all.

Using quotes to include

spaces and characters in
filenames

If you want to work with files with spaces or special characters in the filename, you
may have to use quotes. For instance, if you wanted to create a file with a space in the
name, you could use the following:

% cp /dev/null 'a file with spaces in the name'

Normally, the shell uses spaces to determine the end of each argument. Quoting
changes that, and the above example only has two arguments. You can also use a
backslash before the character. The example below will rename a file with a space in
the name, changing the space to an underscore:

% mv a\ file a_file

Using the same techniques, you can deal with any character in a filename:

% mv a 'a?'

At worst, a space in a file makes it difficult to use as an argument. Other characters are
very dangerous to use in a filename. In particular, using "?" and "*" in a filename is
playing with fire. If you want to delete the file "a?" you may end up deleting more than
the single file.
 Quotes within Quotes

While having two types of quotes (three if you count the backslash) might seem
confusing, in reality it provides you with several ways to solve the same problems. You
can put either quotes inside the other. If you want to quote single quotes, use double
quotes around it. To quote double quotes, use single quotes. Heck, it's easier to show
you:

% echo "Don't do that"

Don't do that
% echo 'The quote of the day is: "TGIF"'
The quote of the day is: "TGIF"
%

Finding out if your quotes

are wrong

In some cases, you may need to use the backslash when you are not sure. In other
cases, a backslash will do the wrong thing. How can you find out if you are quoting
things correctly? The answer: use the shell.

An easy way to check quotes is to add an "echo" before the command so you can see
what is happening, or change an "ls" command into an "echo" command:

echo rcp gateway:\*.tar.Z .

rsh -n cruncher echo ls \*
rsh -n cruncher echo 'ls *'

If you want to do file redirection on a remote machine, echo isn't sufficient. The
command

rsh -n cruncher echo 'ls * >/tmp/file'

sends the results of the echo to the file when you wanted the command echoed to your
terminal. You need to nest the quotes:

rsh -n cruncher "echo 'ls * >/tmp/file'"

rsh -n cruncher 'echo "ls * >/tmp/file"'
rsh -n cruncher "echo 'cd newdir;ls * >>/tmp/file'"

If you are debugging a shell script, and you want to see what your script is doing, you
can duplicate one of the important lines in your script and insert an "echo" in front of
one of the duplicates. Doing this one or two times in a script isn't very difficult, but
there are times when you want to watch every line of your script. In this case, just ask
the shell to show you want is going on.
 The verbose and echo
variables in the C shell

The C shell has two variables that, when set, will help you follow the convoluted trail of
variable and meta-character expansion. The command

set verbose

will echo every line of your script before the variables have been evaluated. The
command

set echo

will display each line after the variables and meta-characters have been substituted. If
you wish to turn the variables off, useunset instead ofset

A convenient way to turn these variables on the first line of the script using the the "-x"
option (echo)

#!/bin/csh -x

or the "-v" option (verbose):

#!/bin/csh -v

In both examples above, the .cshrc file is read at the beginning of the script. The "-f"
option can skip this file. You can combine all three options if you like:

#!/bin/csh -fxv

If you want to read in the .cshrc file, and want to trace the values of these variables,
capitalize the "X" and "V" variables. This turns on tracing before the.cshrc file is read:

#!/bin/csh -XV

It is not necessary to modify the program if you want to turn on the verbose orecho
variables. If this is a script that you do not have the permissions to modify, you can set
these variables from the command line:

% csh -x shell_script
 The Bourne Shell variables

You can enable variable expansion to Bourne shell scripts the same way:

% sh -v script
% sh -x script

Inside a Bourne shell script, the syntax is different. To turn on the verbose flag, use

set -v

to turn on the echo variable, use:

set -x

If you want to turn these variables off, use a plus instead of a minus:

set +x
set +v

Including identical quotes

within quotes

One problem people have is including the same quotes within quotes. Many expect the
following to work:

echo "The word for today is \"TGIF\""

echo 'Don\'t quote me'

The first example works with the Bourne shell, but not the C shell. The second example
doesn't work for either of them. I bet many of you programmers are confused by this.
All of us are very familiar with strings in programming languages like C. This is why we
get confused. The quotes turn substitution on and off. They are not used to indicate the
starting and ending of a string. Consider the following set of quotes:

echo 'a'b'c'

This is broken up into three units. The first and last are quoted, and the middle is not.
After quoting and substitution occurs, the three units are combined. The middle can be
a variable, for instance:

echo 'a'$HOME'b'
This technique is a typical way to get a shell variable into an awk script. Here is a
simple shell script that demonstrates this. Please study this, as it is important:

#!/bin/sh
# this is a shell script that acts like a filter,
# but in only prints out one column.
# the value of the column is the argument
# to the script
#
# uncomment the next line to see how this works
#set -x
#
# example:
# printcol 1
# printcol 3
# the value of the argument is $1
# Here comes the tricky part -
awk '{print $'$1'}'
# I told you!

In this example, the shell breaks up the argument to awk into three pieces: pieces:

{print $ Quoted
$1 Evaluated
} Quoted

You can uncomment the command "set -x" and try the script with the command

printcol 2 </etc/hosts

The argument to the shell script is 2, so "$1" is evaluated and returns the value "2."
This makes the argument to awk the string "print {$2}" and the second column is
printed out.

You must understand this when you want to have quotes within quotes. In fact, you
don't want to put quotes within quotes, you want to combine or concatenate several
units into one argument.

Let me rephrase that. If you want to include a single quote in an argument that starts
with a single quote, you must turn off the mechanism started by the single quote, and
use a different quoting method. Remember, the backslash is the strongest of all quoting
mechanisms. You can quote anything with the backslash. This example quotes all three
quote characters:

% echo \'\"\\

Where the results are

'"\

You can always use the backslash to quote a character. However, within the single
quote mechanism, "\'" does not "quote the quote." The proper way to do this is as
follows:

% echo 'Don' \' 't do that'

Don ' t do that

I put in a few extra spaces, so you could follow what was happening. Here it is again
without the extra spaces:

% echo 'Don'\''t do that'

Don't do that

Just remember to match the quotes together when you mentally parse a shell script.
This also works with double quotes:

% echo "The quote for today is "\"TGIF\"

The quote for today is "TGIF"

Or, if you want to put quotes around "TGIF:"

% echo "The quote for today is "\""TGIF"\"

The quote for today is "TGIF"

Note that this technique works for any shell using either one of the quotes.

Quoting long lines

Most Unix programs also use the backslash to escape a special characters. It is
common for a Unix utility to interpret a backslash on the end of a line as a line
continuation character--that is, the end of line character is quoted or escaped so the
standard meaning is prevented.

The Bourne shell and C shell behave differently when you are quoting lines that
continue beyond the end of line marker. The C shell will not extend a quote beyond the
line unless the last character is a backslash:

% echo "A quote \

on two lines"
A quote
on two lines
The Bourne shell does allow quotes to extend beyond lines:

$ echo "A quote

> on two lines"
A quote
on two lines

Notice how the Bourne shell prompts you with a ">" when the quote is not closed. You
can argue among yourselves which behavior is correct. I can understand the rational
behind the C shell's reason for disallowing multi-line quoting by default. The earlier
example:

echo 'Don\'t do that'

will generate an error if you use the C shell. However, if you use the Bourne shell, you
will be given a ">" prompt and this will continue until you type in a single quote. This
can be confusing for new Unix users.

I find the Bourne shell easier to use when I write multi-line quotes which are awk
scripts:

#!/bin/sh -x
#This script counts how many people
# are in the group specified as the first argument
grp=${1:?"Missing argument"} # get group ID number
# If missing, report an error and exit.
awk -F: '
# Awk script starts here
BEGIN {
# set total to zero
# before we start
total=0;
}
$3 ~ /^'$grp'$/ {total++;}
END {
# end of file, print total
printf("Total: %d\n", total);
}' </etc/passwd

This example uses the "$grp" shell variable right in the middle of the awk script. This is
a common method of passing shell variables into the middle of an awk script.

HERE IS documents

There is another type of quote the shells support. There are calledHere is documents.
There are used when you need to read something from standard input, but you don't
want to create a file to provide that input. There are also used to create files in a shell
script. This can be done by the "<<" character, followed by a special word:
% sort >file <<EndOfSort
zygote
abacus
EndOfSort

This is very useful because variables are evaluated during this operation. Here is a way
to transfer a file usingftp from a shell script:

#!/bin/sh
# Usage:
# ftpfile machine file
# set -x
SOURCE=$1
FILE=$2
BFILE=`basename $FILE`
ftp -n $SOURCE <<EndFTP
ascii
user anonymous $USER@`hostname`
get $FILE /tmp/$BFILE
EndFTP

As you can see, variables and command substitutions are done. If you want the quoted
text to be left alone, put a slash in front of the name of the word:

cat >file <<\FaunkyStriNG

Notice the funky string. This is done just in case the text contains the word that ends
the file. It is very unlikely that I will want to put that particular combination of
characters in any file. You should be warned that the C shell expects the word to be
escaped. i.e. "\FunkyStriNG" while the Bourne shell does not. It expects to see
"FunkyStriNG."