Chapter 8: Searching, Extracting, and Archiving Data

Review Questions: answer in your own words this questions.

1. Which of the following commands will print lines from the file world.txt that
contain matches to changes and changed?

A. grep change[ds] world.txt

B. tar change[d-s] world.txt
C. find "change'd|s'" world.txt
D. cat world.txt changes changed
E. find change[^ds] world.txt

2. Which of the following redirection operators appends a program’s standard output

to an existing file without overwriting that file’s original contents?

A. |
B. 2>
C. &>
D. >
E. >>

3. You’ve received a tar archive called data79.tar from a colleague, but you want to
check the names of the files it contains before extracting them. Which of the following
commands would you use to do this?

A. tar uvf data79.tar

B. tar cvf data79.tar
C. tar xvf data79.tar
D. tar tvf data79.tar
E. tar Avf data79.tar

4. True or false: The regular expression Linu[^x].*lds matches the string Linus
Torvalds .

5. True or false: The find command enables you to locate files based on their sizes.

6. True or false: To compress files archived with zip , you must use an external compression
program such as gzip or bzip2 in a pipeline with zip .

7. The character that represents the start of a line in a regular expression is .

8. The command can extract specified data fields from a file’s records.

9. Complete the following command to redirect both standard output and a standard
error from the bigprog program to the file out.txt .
$ bigprog out.txt

10. The gzip , bzip2 , and xz programs all perform compression, in which
the uncompressed data exactly matches the original precompression data.
Exercise 8.a: Using grep, find, and regular expressions (Objective 3.2)

Linux Distribution: Fedora (non-root user & password needed)

Desktop Environment: GNOME 3

1. If you have not already done so, boot up your computer (or virtual machine) to start
the Fedora Linux distribution.
2. When the machine has booted up, access the tty2 virtual terminal by pressing
Ctrl+Alt+F2.
3. Log on to a regular user’s account by typing in the username at the login prompt and
entering the user account’s password at the Password prompt.
4. Try out the grep command by typing: grep hostname /etc/profile and press
Enter. You should see a single line from the /etc/profile file displayed, similar to:
HOSTNAME = `/bin/hostname 2>/dev/null`
The grep command allows you to search through files and/or directories using
something called "regular expressions." In this case you are just using a character
sequence made up of a series of characters, hostname, to search for the
characters within the file, /etc/profile.
5. In the preceding step, you searched for hostname and all the characters to be
found had to be lowercase. Now have grep ignore the characters' case by typing:
grep -i hostname /etc/profile and press Enter. You should now see two lines from
the /etc/profile file displayed, similar to:
HOSTNAME = `/bin/hostname 2>/dev/null`
export PATH USER LOGNAME MAIL HOSTNAME ...
The -i option tells the grep command to ignore case. Therefore, it will find the
characters you specify in both upper and lower cases.
6. Now you will try out using grep to search for file names that contain the word
hostname and are in the /etc directory.
Type grep -d skip hostname /etc/* and press Enter. Notice that you get both the
filename and the line from the files that contain the word hostname. Don't worry if
you get some "permission denied" messages. Remember, you are logged in as a
regular user and will not have access to several files. This is a great way to search
through configuration files for a setting, if you cannot remember the configuration file
name. The -r option will let you search recursively through various directories.
Exercise 8.a: Using grep, find, and regular expressions (Objective 3.2)
(continued)

7. Try to locate just the root user's password file record by entering the command grep
root /etc/passwd and press Enter. Notice you got two records instead of one,
because the character sequence "root" is embedded within another
/etc/passwd record.
8. This time, you will be able to get grep to locate just the root user's password
file record, by searching for only the record that starts with the character sequence
"root." Type grep ^root /etc/passwd and press Enter. Now you should see only
root's password file record. The ^ metacharacter in front of your character
sequence will only return lines that have that character sequence at the line’s
beginning.
9. Try to locate every record in the hosts file that ends with the character sequence
"local host" by typing grep localhost /etc/hosts and press Enter. Notice you got
two records instead of one, and one of the records does not end with the character
sequence "localhost."
10. This time, you will be able to get grep to locate only the records that end in the
word "localhost" by typing grep localhost$ /etc/hosts and pressing Enter. Now
you should see only one record, and that record actually ends with the word
"localhost." The $ metacharacter at the back of your character sequence will only
return lines that have the character sequence at the line’s end.
11. Using an extended regular expression, you will search for one of two possible
character sequences, your username or root, in the /etc/passwd file. Using
your username for the Username in the command, type grep -E “(Username|root)”
/etc/passwd and press Enter. You should see at least two /etc/passwd file
records containing either your username or root. The -E indicates that an
extended regular expression will be used.
12. Instead of searching for information within a file, now you will be digging out
information about a file. Type wc /etc/hosts and press Enter. You will get three
numbers back; record the numbers you received from the command
here:____________________. The wc command produces output that shows three
statistics for the file: line count, word count, and byte count.
13. Try another file with wc: Type wc /etc/passwd and press Enter. How many lines
are in the /etc/passwd file? Record your answer here:__________. Remember, the
first number returned is the line count.
14. Instead of searching for information about files, you will be searching for files
themselves. Type find /usr/share/doc/ -name COPYING and press Enter. You
should “find” lots of files! Remember that the COPYING files are typically where
software licenses are kept for various Linux software applications. The -name
option of the find command allows you to search for a file by name.
15. Compare the locate command with the find command by typing locate
COPYING and press Enter. Did locate find more files than the find
command? It’s hard to tell, isn’t it! Remember, the locate command searches
though a database of file names and their locations, so it is typically a little quicker
than the find command.
Exercise 8.a: Using grep, find, and regular expressions (Objective 3.2)
(continued)

16. Now wc will help you determine which command (find or locate) is finding
more files, starting with the find command: Type
find /usr/share/doc/ -name COPYING | wc and press Enter. Note: The | symbol
is not a lowercase L or a number one. Instead, it is the pipe symbol and is located on
your keyboard on the \ key. On your keyboard, press Shift and \ to produce the |
symbol (ignore any “permission denied” messages). Record here the first number wc
displays: _________.
17. Type locate COPYING | wc and press Enter. Record here the first number wc
displays: _________. Comparing the results of this step and the previous step, which
command found more files, find or locate? (Actually, this was not a “fair fight”
because the find command was restricted to a single directory location.)
18. Let's take a look at logical operators. First try out this echo command: type echo
"Hello" and press Enter.
19. Logical operators allow you to control what commands are issued. They are more
useful in shell scripts than on the command line. Try a logical operator by typing
echo "Hello" && echo "2nd Hello" and pressing Enter. Both echo commands
should have executed and shown their messages to the screen, like so:
Hello
2nd Hello
This happens because the && symbols are logical operations. When the first
command finishes, then the second command will execute (run).
20. Now see what happens when the first command does not work. Type the following:
cat /etc/PASSWORD && echo "2nd Hello" and press Enter. Notice that the file
name /etc/passwd, is misspelled. This is on purpose. You should see that
neither command properly executed. This is because if the first command doesn't
work, the second command will not be run, due to the && operator. && is a logical
"And."
21. Try another logical operator and see its effect on the display outcome, by typing cat
/etc/PASSWORD || echo "2nd Hello" and pressing Enter. The || symbols are
not lowercase Ls or number ones. Instead they are the pipe symbols. On your
keyboard, press Shift and \ two times to produce the || symbols. You should see
that the second command did execute (run), even though the first command failed.
The || symbol is a logical "Or."
22. Try two potentially successful commands by typing
echo "Hello" || echo "2nd Hello" and pressing Enter. Because the || is a logical
"Or" and since the first command executed successfully, it did not run the second
command.
Exercise 8.b: Exploring Redirection and Pipes (Objective 3.2)

Linux Distribution: Fedora (non-root user & password needed)

Desktop Environment: GNOME 3

1. If you have not already done so, boot up your computer (or virtual machine) to start
the Fedora Linux distribution.
2. When the machine has booted up, access the tty2 virtual terminal by pressing
Ctrl+Alt+F2.
3. Log on to a regular user’s account by typing in the username at the login prompt and
entering the user account’s password at the Password prompt.
4. Type echo "Hello" and press Enter. The word Hello should have displayed to
your computer screen (Standard Output or STDOUT).
5. Try redirecting STDOUT to a file by typing echo "Hello" > new_file and pressing
Enter. Notice nothing was displayed on the screen! This is because standard output,
which normally displays to the screen, was redirected (using the >) into a file.The
word Hello should now be stored in the file new_file.
6. See what is in the file new_file by typing cat new_file and pressing Enter. You
should see the word Hello, because the word was redirected into the file in the
previous step by using the > redirection operator.
7. Redirect output again by typing cat new_file > second_file and pressing Enter.
8. See if the output redirected in the step above is in the file second_file by
typing cat second_file and pressing Enter. The word Hello should have
displayed to your computer screen.
9. See what happens to the original contents of new_file when additional data is
redirected into it. Type echo "I am redirected" > new_file and press Enter. This
will redirect the output of "I am redirected" into the file new_file instead of to the
screen.
10. Now see if the redirect above worked by typing cat new_file and pressing Enter.
Using redirect causes the file's original contents to be wiped and replaced by what
you redirected into it.
11. You can redirect standard output and append the output to a file, using the symbols
>> together. Try appending to the file by typing
echo "I appended this" >> new_file and pressing Enter.
12. Now see if the redirect above worked by typing cat new_file and pressing Enter.
Using two redirection symbols, >>, appends the output to the end of a file.
13. So far you have just been redirecting output. Now you will try redirecting Standard
Error (STDERR). First, take a look at where error messages are directed to by
default, by typing a command that won't work. Type
cat /etc/PASSWORD and press Enter. Note: The file /etc/PASSWORD does
not exist and therefore will generate an error message. By default, error messages
are displayed to your terminal screen.

Continued on the next page

Exercise 8.b: Exploring Redirection and Pipes (Objective 3.2) (continued)

14. Now, try redirecting the error message by typing

cat /etc/PASSWORD 2>> new_file and pressing Enter.
There are two things to notice in this command: 1) The number 2 was used to
redirect the error message. You did not have to put a number in front of the two
redirection symbols (>>) when redirecting output earlier. However, redirecting error
messages requires a 2.
2) Since two redirection symbols (>>) were used instead of one, the error message
will be appended to the end of the file new_file.
15. Take a look at how the file new_file looks now by typing cat new_file and
pressing Enter. The error message should now be appended at the end of
new_file's contents.
16. Now try "wiping out" the contents of new_file and just have an error message in
the file, by using only one redirection symbol. Type
cat /etc/PASSWORD 2> new_file and press Enter. Remember, just using a single
redirection symbol will wipe out the current contents of the file and put in the new
contents. In this case, the new contents will be an error message.
17. See if your error message was properly redirected by typing cat new_file and
pressing Enter.
18. Before you start exploring pipes, clean up the two files you used in redirecting output
and an error message: type rm -i *file and press Enter. Type y and press Enter for
the questions concerning deleting these two files.
19. Create some new files in your pwd by typing
touch fileA.dat fileB.dat fileC.dat fileD.dat and pressing Enter.
20. Make sure all four files were created by typing ls file?.dat and pressing Enter. Create
any files you missed in the preceding step.
21. Use the pipe symbol to display the files in alphabetically sorted order. The | (pipe)
symbol can appear either as a solid line or as a vertical dash. On your keyboard,
press Shift and \ to produce the | symbol. A pipe redirects the standard output of the
first command into input for the second command. Try it out by typing ls file?.dat |
sort and pressing Enter. That’s not too interesting, because files are typically shown
in alphabetically sorted order.
22. Reverse the sort order by typing ls file?.dat | sort -r and pressing Enter. The pipe
symbol took the ls command’s standard output and sent it as standard input into
the sort command, which then reverse-sorted the file names and displayed them
to standard output (the screen).
23. You can display piped output and save it to a file using the tee command. Try this
out by typing ls file?.dat | sort -r | tee fileE.dat and pressing Enter.
24. See if the output was saved to the fileE.dat file by typing cat fileE.dat and
pressing Enter. You will see that it did save the output. The tee command can be
handy when employing pipes.

Continued on the next page

Exercise 8.b: Exploring Redirection and Pipes (Objective 3.2) (continued)

25. Instead of doing a regular exercise file cleanup with the rm command, you will use
the xargs command. The xargs command will take information from standard
output piped to it, build a command from it, and execute it. Type ls file?.dat | xargs
/bin/rm and press Enter. It looks like nothing happened, but all the file?.dat
files were deleted. The xargs commands uses an absolute directory reference for
the commands it builds. That’s why /bin/rm was used, instead of simply rm.
26. See if the files were deleted by typing ls file?.dat and pressing Enter. You should
receive a “No such file or directory” message or something similar.

Exercise 8.c: Trying Out Compression and Archiving (Objective 3.1)

Linux Distribution: Fedora (non-root user & password needed)

Desktop Environment: GNOME 3

1. If you have not already done so, boot up your computer (or virtual machine) to start
the Fedora Linux distribution.
2. When the machine has booted up, access the tty2 virtual terminal by pressing
Ctrl+Alt+F2.
3. Log on to a regular user’s account by typing in the username at the login prompt and
entering the user account’s password at the Password prompt.
4. Create some files to use in this exercise: at the prompt, type
touch file_a.log file_b.log file_c.log file_d.log and press Enter.
5. To ensure the files have been created, type ls *.log and press Enter. Create any
files you missed in the preceding step.
6. Compress the files using gzip by typing gzip file*.log and pressing Enter. This
will compress each file into a new file (ending in .gz) and delete the original file.
7. Type ls file*.gz and press Enter to see the newly compressed files.
8. Uncompress the files using gzip by typing gunzip file*.gz and pressing Enter.
This does the exact opposite of step #6. It uncompresses each file to its original
name and deletes the .gz file.
9. Type ls file* and press Enter to see the uncompressed files.
10. Now try compressing the files using bzip2 by typing bzip2 file*.log and pressing
Enter. This does the exact same thing as gzip except the compression is slightly
higher and the files end in .bz2.
11. Type ls file*.bz2 and press Enter to see the newly compressed files. Now try
uncompressing the files using bzip2 by typing bunzip2 file*.bz2 and pressing
Enter. This does the exact opposite of step #10. It uncompresses each file to its
original name and deletes the .bz2 file. This is the way most compression utilities
work, including xz and zip (to uncompress those two, you use unxz and
unzip).
Continued on the next page
Exercise 8.c: Trying Out Compression and Archiving (Objective 3.1)
(continued)

12. Now you will create a single file from a group of files. The files will exist within the
single file. Type tar cvf file.tar file*.log and press Enter. All of the individual files
are copied and put into a "box" of a single file that you named file.tar. The c
option creates the tar file. The v option tells you what is going on (be verbose). The
f option indicates what you want to name the file (in this case file.tar).
Finally, you tell the tar command what files you want to include in this "box."
13. Type ls file*.* and press Enter. You should see all the original files and the
file.tar file. Now you have a backup copy of the file*.log files and
they are stored in the file.tar file. This is called an archive file.
14. Type tar zcvf file.tar.gz file*.log and press Enter. This will create a tar file and
compress it. A compressed archive file is called a tarball. The z option compresses
the tar file into a tarball using gzip compression.
15. Type ls file*.* and press Enter. You should see all the original files, the
file.tar tar file, and the tarball file.tar.gz.
16. Create a new directory by typing mkdir unpack_tar and pressing Enter. This is a
new directory where you will unpack the tarball.
17. Move the gzip tarball to the new directory by typing
mv file.tar.gz unpack_tar/ and pressing Enter. Notice that you used a relative
directory reference when you moved the tarball.
18. Go to the new directory by typing cd unpack_tar and pressing Enter.
19. Type ls *.tar.gz and press Enter. This will let you check and see if the tarball is in the
new directory. You should see the tarball in this directory.
20. Uncompress and unpack the tarball by typing tar zxvf file.tar.gz and pressing
Enter. You replace the c option with the x option to unpack the tarball. The z option
is versatile, in that it is used to compress or uncompress the tar file.
21. Type ls and press Enter. You should see both the tarball file and all the files you
originally put into it.
22. Go back to your home directory. Type cd and press Enter.
23. Remove the unpacked directory and its contents by typing
rm -ir unpack_tar and pressing Enter. Type y and press Enter for every question
on deleting the files you moved and unpacked in that directory, including the directory
itself.
24. Remove the files you originally created for compression and archiving by typing rm -i
file*.* and pressing Enter. Type y and press Enter for every question on deleting the
files you used in compression and archiving.