Experience Seamless Full Ebook Downloads for Every Genre at textbookfull.

com

How Linux Works What Every Superuser Should Know

3rd Edition Brian Ward

https://textbookfull.com/product/how-linux-works-what-every-
superuser-should-know-3rd-edition-brian-ward-2/

OR CLICK BUTTON

DOWNLOAD NOW

Explore and download more ebook at https://textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

How Linux Works What Every Superuser Should Know 3rd

Edition Brian Ward

https://textbookfull.com/product/how-linux-works-what-every-superuser-
should-know-3rd-edition-brian-ward-2/

textboxfull.com

How Linux Works What Every Superuser Should Know Brian

Ward

https://textbookfull.com/product/how-linux-works-what-every-superuser-
should-know-brian-ward/

textboxfull.com

How Linux Works 3rd Edition Brian Ward

https://textbookfull.com/product/how-linux-works-3rd-edition-brian-
ward/

textboxfull.com

What Every Science Student Should Know Justin Bauer

https://textbookfull.com/product/what-every-science-student-should-
know-justin-bauer/

textboxfull.com
What Every Engineer Should Know About Excel Jack P. Holman

https://textbookfull.com/product/what-every-engineer-should-know-
about-excel-jack-p-holman/

textboxfull.com

Vinod Pottayil What Every Indian Should Know Before

Investing First Edition Vinod Pottayil

https://textbookfull.com/product/vinod-pottayil-what-every-indian-
should-know-before-investing-first-edition-vinod-pottayil/

textboxfull.com

What Every Engineer Should Know About Modeling and

Simulation 1st Edition Raymond J. Madachy

https://textbookfull.com/product/what-every-engineer-should-know-
about-modeling-and-simulation-1st-edition-raymond-j-madachy/

textboxfull.com

What Every Engineer Should Know about Reliability and Risk

Analysis First Edition Mohammad Modarres

https://textbookfull.com/product/what-every-engineer-should-know-
about-reliability-and-risk-analysis-first-edition-mohammad-modarres/

textboxfull.com

What Every Woman Should Know about Cervical Cancer Revised

and Updated 2nd Edition Nenad Markovic

https://textbookfull.com/product/what-every-woman-should-know-about-
cervical-cancer-revised-and-updated-2nd-edition-nenad-markovic/

textboxfull.com
 3RD EDITION

HOW
LINUX
WORKS
W H A T E V E R Y S U P E R U S E R S H O U L D K N O W

BRIAN WARD
 REVIEWS FOR
HOW LINUX WORKS

“If you are interested in Linux, How Linux Works: What Every Superuser Should
Know is a must-read title.”
—LinuxInsider

“Lots to offer on almost every aspect of the Linux architecture.”

—Everyday Linux User

“You’ll get an essential understanding of what’s going on under the hood with-
out getting bogged down in minutiae—making this a very refreshing (and
wholly recommended) addition to the Linux literature.”
—Phil Bull, co-author of Ubuntu
M ade Easy and member of the
Ubuntu documentation team

“Dives straight into the transparent depths of Linux-based operating systems

and shows us how all the pieces fit together.”
—DistroWatch

“Earns its place on the shelf as an essential reference.”

—The M agPi magazine
HOW LINUX
WORKS
3rd Edition
What Every Superuser
Should Know

b y Br i a n Wa r d

San Francisco
HOW LINUX WORKS, 3RD EDITION. Copyright © 2021 by Brian Ward.

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, recording, or by any information storage or retrieval
system, without the prior written permission of the copyright owner and the publisher.

ISBN-13: 978-1-7185-0040-2 (print)

ISBN-13: 978-1-7185-0041-9 (ebook)

Publisher: William Pollock

Executive Editor: Barbara Yien
Production Editor: Rachel Monaghan
Developmental Editors: Larry Wake and Jill Franklin
Cover and Interior Design: Octopod Studios
Technical Reviewers: Jordi Gutiérrez Hermoso and Petros Koutoupis
Copyeditor: Rachel Monaghan
Compositor: Cody Gates, Happenstance Type-O-Rama
Proofreader: James M. Fraleigh

For information on book distributors or translations, please contact No Starch Press, Inc. directly:
No Starch Press, Inc.
245 8th Street, San Francisco, CA 94103
phone: 1-415-863-9900; [email protected]
www.nostarch.com

The Library of Congress has catalogued the first edition as follows:

Ward, Brian.
How Linux works : what every superuser should know / Brian Ward.
p. cm.
Includes index.
ISBN 1-59327-035-6
1. Linux. 2. Operating systems (Computers). I. Title.
QA76.76.O63 W3654 2004
005.4’32--dc22
2004002692

No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other
product and company names mentioned herein may be the trademarks of their respective owners. Rather
than use a trademark symbol with every occurrence of a trademarked name, we are using the names only
in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of
the trademark.

The information in this book is distributed on an “As Is” basis, without warranty. While every precaution
has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any
liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly
or indirectly by the information contained in it.
About the Author
Brian Ward has been working with Linux since 1993. He is also the
author of The Linux Kernel HOWTO, The Book of VMware (No Starch
Press), and The Linux Problem Solver (No Starch Press).

About the Technical Reviewers

Jordi Gutiérrez Hermoso is a GNU/Linux user and developer with
almost two decades of experience, with occasional contributions in vari-
ous circles of the free software communities, such as GNU Octave and
Mercurial. His professional work has allowed him to collaborate with
and learn about diverse topics such as digital cryptographic signatures,
medical imaging, and most recently greenhouse gas accounting and
ecological map data, all built entirely with Linux and other free soft-
ware. When he is not near a computer he enjoys swimming, mathematics,
and knitting.

Petros Koutoupis is currently a senior performance software engineer

at HPE (formerly Cray Inc.) for its Lustre High Performance File System
division. He is also the creator and maintainer of the RapidDisk Project
(www.rapiddisk.org). Petros has worked in the data storage industry
for well over a decade and has helped pioneer the many technologies
unleashed in the wild today.
 BRIEF CONTENTS

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi

Chapter1: The Big Picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 2: Basic Commands and Directory Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Chapter 3: Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Chapter 4: Disks and Filesystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Chapter 5: How the Linux Kernel Boots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Chapter 6: How User Space Starts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Chapter 7: System Configuration: Logging, System Time, Batch Jobs, and Users . . . . . . . . . 167

Chapter 8: A Closer Look at Processes and Resource Utilization . . . . . . . . . . . . . . . . . . . . 199

Chapter 9: Understanding Your Network and Its Configuration . . . . . . . . . . . . . . . . . . . . . 223

Chapter 10: Network Applications and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

Chapter 11: Introduction to Shell Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

Chapter 12: Network File Transfer and Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Chapter 13: User Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335

Chapter 14: A Brief Survey of the Linux Desktop and Printing . . . . . . . . . . . . . . . . . . . . . . 347

Chapter 15: Development Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363

Chapter 16: Introduction to Compiling Software from C Source Code . . . . . . . . . . . . . . . . 385

Chapter 17: Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
 CO N T E N T S I N D E TA I L

PREFACE XXI
Who Should Read This Book? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
How to Read This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
A Hands-on Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
How This Book Is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
What’s New in the Third Edition? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
A Note on Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv

1
THE BIG PICTURE 1
1.1 Levels and Layers of Abstraction in a Linux System . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Hardware: Understanding Main Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 The Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.1 Process Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.2 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.3 Device Drivers and Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.4 System Calls and Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 User Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6 Looking Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2
BASIC COMMANDS AND DIRECTORY HIERARCHY 11
2.1 The Bourne Shell: /bin/sh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Using the Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 The Shell Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.2 cat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.3 Standard Input and Standard Output . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Basic Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.1 ls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.2 cp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.3 mv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.4 touch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.5 rm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.6 echo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Navigating Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.1 cd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.2 mkdir . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.3 rmdir . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.4 Shell Globbing (“Wildcards”) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.5 Intermediate Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5.1 grep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5.2 less . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5.3 pwd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5.4 diff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.5.5 file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.5.6 find and locate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.5.7 head and tail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.5.8 sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.6 Changing Your Password and Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.7 Dot Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.8 Environment and Shell Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.9 The Command Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.10 Special Characters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.11 Command-Line Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.12 Text Editors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.13 Getting Online Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.14 Shell Input and Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.14.1 Standard Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.14.2 Standard Input Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.15 Understanding Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.15.1 Anatomy of a Unix Error Message . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.15.2 Common Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.16 Listing and Manipulating Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.16.1 Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.16.2 Process Termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.16.3 Job Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.16.4 Background Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.17 File Modes and Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.17.1 Modifying Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.17.2 Working with Symbolic Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.18 Archiving and Compressing Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.18.1 gzip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.18.2 tar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.18.3 Compressed Archives (.tar.gz) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.18.4 zcat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.18.5 Other Compression Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.19 Linux Directory Hierarchy Essentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.19.1 Other Root Subdirectories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.19.2 The /usr Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.19.3 Kernel Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.20 Running Commands as the Superuser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.20.1 sudo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.20.2 /etc/sudoers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.20.3 sudo Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.21 Looking Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3
DEVICES 47
3.1 Device Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2 The sysfs Device Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

x Contents in Detail
3.3 dd and Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.4 Device Name Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.4.1 Hard Disks: /dev/sd* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.4.2 Virtual Disks: /dev/xvd*, /dev/vd* . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4.3 Non-Volatile Memory Devices: /dev/nvme* . . . . . . . . . . . ..... . . . 53
3.4.4 Device Mapper: /dev/dm-*, /dev/mapper/* . . . . . . . . . . . . . . . . . . 53
3.4.5 CD and DVD Drives: /dev/sr* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4.6 PATA Hard Disks: /dev/hd* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4.7 Terminals: /dev/tty*, /dev/pts/*, and /dev/tty . . . . . . . . . . . . . . . . . 53
3.4.8 Serial Ports: /dev/ttyS*, /dev/ttyUSB*, /dev/ttyACM* . . . . . . . . . . . 55
3.4.9 Parallel Ports: /dev/lp0 and /dev/lp1 . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.10 Audio Devices: /dev/snd/*, /dev/dsp, /dev/audio, and More . . . . 55
3.4.11 Device File Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5 udev . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5.1 devtmpfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.5.2 udevd Operation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.5.3 udevadm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.5.4 Device Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.6 In-Depth: SCSI and the Linux Kernel . . . . . . . . . . . . . . . . . . . . . . . . . ..... . . . 62
3.6.1 USB Storage and SCSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.6.2 SCSI and ATA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.6.3 Generic SCSI Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.6.4 Multiple Access Methods for a Single Device . . . . . . . . . . . . . . . . . . . 67

4
DISKS AND FILESYSTEMS 69
4.1 Partitioning Disk Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.1.1 Viewing a Partition Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.1.2 Modifying Partition Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.1.3 Creating a Partition Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.1.4 Navigating Disk and Partition Geometry . . . . . . . . . . . . . . . . . . . . . . 78
4.1.5 Reading from Solid-State Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2 Filesystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2.1 Filesystem Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.2.2 Creating a Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.3 Mounting a Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.2.4 Filesystem UUID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2.5 Disk Buffering, Caching, and Filesystems . . . . . . . . . . . . . . . . . . . . . . 86
4.2.6 Filesystem Mount Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.2.7 Remounting a Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.2.8 The /etc/fstab Filesystem Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.2.9 Alternatives to /etc/fstab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.2.10 Filesystem Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.2.11 Checking and Repairing Filesystems . . . . . . . . . . . . . . . . . . . . . . . . 91
4.2.12 Special-Purpose Filesystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.3 Swap Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.3.1 Using a Disk Partition as Swap Space . . . . . . . . . . . . . . . . . . . . . . . . 94
4.3.2 Using a File as Swap Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.3.3 Determining How Much Swap You Need . . . . . . . . . . . . . . . . . . . . . . 95
4.4 The Logical Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.4.2 Working with LVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.4.3 The LVM Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Contents in Detail  xi
4.5 Looking Forward: Disks and User Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.6 Inside a Traditional Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.6.1 Inode Details and the Link Count . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.6.2 Block Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.6.3 Working with Filesystems in User Space . . . . . . . . . . . . . . . . . . . . . 115

5
HOW THE LINUX KERNEL BOOTS 117
5.1 Startup Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.2 Kernel Initialization and Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.3 Kernel Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.4 Boot Loaders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.4.1 Boot Loader Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.4.2 Boot Loader Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.5 GRUB Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.5.1 Exploring Devices and Partitions with the GRUB Command Line . . . . . 125
5.5.2 GRUB Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.5.3 GRUB Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5.6 UEFI Secure Boot Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
5.7 Chainloading Other Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.8 Boot Loader Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.8.1 MBR Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5.8.2 UEFI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5.8.3 How GRUB Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

6
HOW USER SPACE STARTS 137
6.1 Introduction to init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.2 Identifying Your init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.3 systemd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.3.1 Units and Unit Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.3.2 Booting and Unit Dependency Graphs . . . . . . . . . . . . . . . . . . . . . . 140
6.3.3 systemd Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.3.4 systemd Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.3.5 systemd Process Tracking and Synchronization . . . . . . . . . . . . . . . . . 147
6.3.6 systemd Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
6.3.7 systemd On-Demand and Resource-Parallelized Startup . . . . . . . . . . . 151
6.3.8 systemd Auxiliary Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6.4 System V Runlevels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6.5 System V init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
6.5.1 System V init: Startup Command Sequence . . . . . . . . . . . . . . . . . . . 158
6.5.2 The System V init Link Farm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6.5.3 run-parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
6.5.4 System V init Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.5.5 systemd System V Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.6 Shutting Down Your System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
6.7 The Initial RAM Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
6.8 Emergency Booting and Single-User Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
6.9 Looking Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

xii Contents in Detail

7
SYSTEM CONFIGURATION:
LOGGING, SYSTEM TIME, BATCH JOBS, AND USERS 167
7.1 System Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
7.1.1 Checking Your Log Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
7.1.2 Searching and Monitoring Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
7.1.3 Logfile Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
7.1.4 Journal Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.1.5 A Closer Look at System Logging . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.2 The Structure of /etc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
7.3 User Management Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.3.1 The /etc/passwd File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.3.2 Special Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
7.3.3 The /etc/shadow File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
7.3.4 Manipulating Users and Passwords . . . . . . . . . . . . . . . . . . . . . . . . . 179
7.3.5 Working with Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.4 getty and login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.5 Setting the Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.5.1 Kernel Time Representation and Time Zones . . . . . . . . . . . . . . . . . . . 182
7.5.2 Network Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
7.6 Scheduling Recurring Tasks with cron and Timer Units . . . . . . . . . . . . . . . . . . . . . 183
7.6.1 Installing Crontab Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
7.6.2 System Crontab Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7.6.3 Timer Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7.6.4 cron vs. Timer Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
7.7 Scheduling One-Time Tasks with at . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
7.7.1 Timer Unit Equivalents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
7.8 Timer Units Running as Regular Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
7.9 User Access Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.9.1 User IDs and User Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.9.2 Process Ownership, Effective UID, Real UID, and Saved UID . . . . . . . 189
7.9.3 User Identification, Authentication, and Authorization . . . . . . . . . . . . 191
7.9.4 Using Libraries for User Information . . . . . . . . . . . . . . . . . . . . . . . . . 192
7.10 Pluggable Authentication Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
7.10.1 PAM Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
7.10.2 Tips on PAM Configuration Syntax . . . . . . . . . . . . . . . . . . . . . . . . 196
7.10.3 PAM and Passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
7.11 Looking Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

8
A CLOSER LOOK AT PROCESSES AND RESOURCE UTILIZATION 199
8.1 Tracking Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
8.2 Finding Open Files with lsof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
8.2.1 Reading the lsof Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
8.2.2 Using lsof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
8.3 Tracing Program Execution and System Calls . . . . . . . . . . . . . . . . . . . . . . . . . . 202
8.3.1 strace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
8.3.2 ltrace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
8.4 Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
8.4.1 Single-Threaded and Multithreaded Processes . . . . . . . . . . . . . . . . . 204
8.4.2 Viewing Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205


Contents in Detail xiii
8.5 Introduction to Resource Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
8.5.1 Measuring CPU Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
8.5.2 Adjusting Process Priorities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
8.5.3 Measuring CPU Performance with Load Averages . . . . . . . . . . . . . . . 208
8.5.4 Monitoring Memory Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
8.5.5 Monitoring CPU and Memory Performance with vmstat . . . . . . . . . . . 212
8.5.6 I/O Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
8.5.7 Per-Process Monitoring with pidstat . . . . . . . . . . . . . . . . . . . . . . . . . 216
8.6 Control Groups (cgroups) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
8.6.1 Differentiating Between cgroup Versions . . . . . . . . . . . . . . . . . . . . . 217
8.6.2 Viewing cgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
8.6.3 Manipulating and Creating cgroups . . . . . . . . . . . . . . . . . . . . . . . . 220
8.6.4 Viewing Resource Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
8.7 Further Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

9
UNDERSTANDING YOUR NETWORK AND ITS CONFIGURATION 223
9.1 Network Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
9.2 Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
9.3 Network Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
9.4 The Internet Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
9.4.1 Viewing IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
9.4.2 Subnets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
9.4.3 Common Subnet Masks and CIDR Notation . . . . . . . . . . . . . . . . . . . 229
9.5 Routes and the Kernel Routing Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
9.6 The Default Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
9.7 IPv6 Addresses and Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
9.7.1 Viewing IPv6 Configuration on Your System . . . . . . . . . . . . . . . . . . . 232
9.7.2 Configuring Dual-Stack Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 233
9.8 Basic ICMP and DNS Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
9.8.1 ping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
9.8.2 DNS and host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
9.9 The Physical Layer and Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
9.10 Understanding Kernel Network Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
9.11 Introduction to Network Interface Configuration . . . . . . . . . . . . . . . . . . . . . . . 237
9.11.1 Manually Configuring Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . 237
9.11.2 Manually Adding and Deleting Routes . . . . . . . . . . . . . . . . . . . . . . 238
9.12 Boot-Activated Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
9.13 Problems with Manual and Boot-Activated Network Configuration . . . . . . . . . . 239
9.14 Network Configuration Managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
9.14.1 NetworkManager Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
9.14.2 NetworkManager Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
9.14.3 NetworkManager Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 242
9.15 Resolving Hostnames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
9.15.1 /etc/hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
9.15.2 resolv.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
9.15.3 Caching and Zero-Configuration DNS . . . . . . . . . . . . . . . . . . . . . . 245
9.15.4 /etc/nsswitch.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
9.16 Localhost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
9.17 The Transport Layer: TCP, UDP, and Services . . . . . . . . . . . . . . . . . . . . . . . . . 247
9.17.1 TCP Ports and Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
9.17.2 UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250

xiv Contents in Detail

9.18 Revisiting a Simple Local Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
9.19 Understanding DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
9.19.1 Linux DHCP Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
9.19.2 Linux DHCP Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
9.20 Automatic IPv6 Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
9.21 Configuring Linux as a Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
9.22 Private Networks (IPv4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
9.23 Network Address Translation (IP Masquerading) . . . . . . . . . . . . . . . . . . . . . . . 256
9.24 Routers and Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
9.25 Firewalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
9.25.1 Linux Firewall Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
9.25.2 Setting Firewall Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
9.25.3 Firewall Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
9.26 Ethernet, IP, ARP, and NDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
9.27 Wireless Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
9.27.1 iw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
9.27.2 Wireless Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
9.28 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

10
NETWORK APPLICATIONS AND SERVICES 269
10.1 The Basics of Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
10.2 A Closer Look . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
10.3 Network Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
10.3.1 Secure Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
10.3.2 The sshd Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
10.3.3 fail2ban . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
10.3.4 The SSH Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
10.4 Pre-systemd Network Connection Servers: inetd/xinetd . . . . . . . . . . . . . . . . . . . . . 278
10.5 Diagnostic Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
10.5.1 lsof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
10.5.2 tcpdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
10.5.3 netcat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
10.5.4 Port Scanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
10.6 Remote Procedure Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
10.7 Network Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
10.7.1 Typical Vulnerabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
10.7.2 Security Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
10.8 Looking Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
10.9 Network Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
10.10 Unix Domain Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

11
INTRODUCTION TO SHELL SCRIPTS 291
11.1 Shell Script Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
11.1.1 Limitations of Shell Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
11.2 Quoting and Literals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
11.2.1 Literals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
11.2.2 Single Quotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
11.2.3 Double Quotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
11.2.4 Literal Single Quotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295

Contents in Detail xv
11.3 Special Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
11.3.1 Individual Arguments: $1, $2, and So On . . . . . . . . . . . . . . . . . . . 296
11.3.2 Number of Arguments: $# . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
11.3.3 All Arguments: $@ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
11.3.4 Script Name: $0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
11.3.5 Process ID: $$ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
11.3.6 Exit Code: $? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
11.4 Exit Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
11.5 Conditionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
11.5.1 A Workaround for Empty Parameter Lists . . . . . . . . . . . . . . . . . . . . 299
11.5.2 Other Commands for Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
11.5.3 elif . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
11.5.4 Logical Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
11.5.5 Testing Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
11.5.6 case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
11.6 Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
11.6.1 for Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
11.6.2 while Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
11.7 Command Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
11.8 Temporary File Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
11.9 Here Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
11.10 Important Shell Script Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
11.10.1 basename . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
11.10.2 awk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
11.10.3 sed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
11.10.4 xargs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
11.10.5 expr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
11.10.6 exec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
11.11 Subshells . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
11.12 Including Other Files in Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
11.13 Reading User Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
11.14 When (Not) to Use Shell Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312

12
NETWORK FILE TRANSFER AND SHARING 315
12.1 Quick Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
12.2 rsync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
12.2.1 Getting Started with rsync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
12.2.2 Making Exact Copies of a Directory Structure . . . . . . . . . . . . . . . . . 318
12.2.3 Using the Trailing Slash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
12.2.4 Excluding Files and Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
12.2.5 Checking Transfers, Adding Safeguards,
and Using Verbose Mode . . . . . . . . . . . . . . . . . . . . . . . ...... 321
12.2.6 Compressing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
12.2.7 Limiting Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
12.2.8 Transferring Files to Your Computer . . . . . . . . . . . . . . . . . . . . . . . . 322
12.2.9 Further rsync Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
12.3 Introduction to File Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
12.3.1 File Sharing Usage and Performance . . . . . . . . . . . . . . . . ...... 323
12.3.2 File Sharing Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

xvi Contents in Detail

12.4 Sharing Files with Samba . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
12.4.1 Server Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
12.4.2 Server Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
12.4.3 Passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
12.4.4 Manual Server Startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
12.4.5 Diagnostics and Logfiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
12.4.6 File Share Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
12.4.7 Home Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
12.4.8 Printer Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
12.4.9 The Samba Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
12.5 SSHFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
12.6 NFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
12.7 Cloud Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
12.8 The State of Network File Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333

13
USER ENVIRONMENTS 335
13.1 Guidelines for Creating Startup Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
13.2 When to Alter Startup Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
13.3 Shell Startup File Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
13.3.1 The Command Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
13.3.2 The Manual Page Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
13.3.3 The Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
13.3.4 Aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
13.3.5 The Permissions Mask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
13.4 Startup File Order and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
13.4.1 The bash Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
13.4.2 The tcsh Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
13.5 Default User Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
13.5.1 Shell Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
13.5.2 Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
13.5.3 Pager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
13.6 Startup File Pitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
13.7 Further Startup Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345

14
A BRIEF SURVEY OF THE LINUX DESKTOP AND PRINTING 347
14.1 Desktop Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
14.1.1 Framebuffers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
14.1.2 The X Window System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
14.1.3 Wayland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
14.1.4 Window Managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
14.1.5 Toolkits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
14.1.6 Desktop Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
14.1.7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
14.2 Are You Running Wayland or X? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
14.3 A Closer Look at Wayland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
14.3.1 The Compositing Window Manager . . . . . . . . . . . . . . . . . . . . . . . 351
14.3.2 libinput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
14.3.3 X Compatibility in Wayland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353


Contents in Detail xvii
14.4 A Closer Look at the X Window System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354
14.4.1 Display Managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
14.4.2 Network Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
14.4.3 Ways of Exploring X Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
14.4.4 X Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
14.4.5 X Input and Preference Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
14.5 D-Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
14.5.1 System and Session Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
14.5.2 D-Bus Message Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
14.6 Printing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
14.6.1 CUPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
14.6.2 Format Conversion and Print Filters . . . . . . . . . . . . . . . . . . . . . . . . 361
14.7 Other Desktop Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362

15
DEVELOPMENT TOOLS 363
15.1 The C Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
15.1.1 Compiling Multiple Source Files . . . . . . . . . . . . . . . . . . . . . . . . . . 365
15.1.2 Linking with Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
15.1.3 Working with Shared Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
15.1.4 Working with Header (Include) Files and Directories . . . . . . . . . . . . 371
15.2 make . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
15.2.1 A Sample Makefile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
15.2.2 Built-in Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
15.2.3 Final Program Build . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
15.2.4 Dependency Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
15.2.5 Command-Line Arguments and Options . . . . . . . . . . . . . . . . . . . . . 376
15.2.6 Standard Macros and Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 377
15.2.7 Conventional Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
15.2.8 Makefile Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
15.3 Lex and Yacc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
15.4 Scripting Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
15.4.1 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
15.4.2 Perl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
15.4.3 Other Scripting Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
15.5 Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
15.6 Looking Forward: Compiling Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383

16
INTRODUCTION TO COMPILING SOFTWARE
FROM C SOURCE CODE 385
16.1 Software Build Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
16.2 Unpacking C Source Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
16.3 GNU Autoconf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
16.3.1 An Autoconf Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
16.3.2 Installation Using a Packaging Tool . . . . . . . . . . . . . . . . . . . . . . . . 390
16.3.3 configure Script Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390
16.3.4 Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
16.3.5 Autoconf Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
16.3.6 Autoconf Logfiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
16.3.7 pkg-config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393

xviii Contents in Detail

16.4 Installation Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
16.4.1 Where to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
16.5 Applying a Patch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
16.6 Troubleshooting Compiles and Installations . . . . . . . . . . . . . . . . . . . . . . . . . . 396
16.6.1 Specific Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
16.7 Looking Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399

17
VIRTUALIZATION 401
17.1 Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
17.1.1 Hypervisors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
17.1.2 Hardware in a Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . 403
17.1.3 Common Uses of Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . 404
17.1.4 Drawbacks of Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . . . 404
17.2 Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
17.2.1 Docker, Podman, and Privileges . . . . . . . . . . . . . . . . . . . . . . . . . . 406
17.2.2 A Docker Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
17.2.3 LXC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
17.2.4 Kubernetes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
17.2.5 Pitfalls of Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
17.3 Runtime-Based Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417

BIBLIOGRAPHY 419

INDEX 423


Contents in Detail xix
 ACKNOW LEDGMENT S

Contributions to this book come from not just those who were involved in
the development process, but also those without whom I wouldn’t know
anything about Linux. These include James Duncan, Douglas N. Arnold,
Bill Fenner, Ken Hornstein, Scott Dickson, Dan Ehrlich, Felix Lee, and
Gregory P. Smith. Previous editions included help from Karol Jurado,
Laurel Chun, Serena Yang, Alison Law, Riley Hoffman, Scott Schwartz,
Dan Sully, Dominique Poulain, Donald Karon, and Gina Steele.
This third edition benefits from the excellent work of Barbara Yien,
Rachel Monaghan, Jill Franklin, Larry Wake, Jordi Gutiérrez Hermoso,
and Petros Koutoupis. As the publisher of No Starch Press, Bill Pollock
has been instrumental in this book since the first edition. And once more,
Hsinju Hsieh has put up with me for another revision.
 PR E FAC E

Your system shouldn’t be a mystery. You should be able to make your soft-
ware do what you want it to do without “magic” incantations or rituals. The
key to attaining this power lies in understanding the fundamentals of what
the software does and how it works, and that’s what this book is all about.
You should never have to fight with a computer.
Linux is a great platform for learning because it doesn’t try to hide
anything from you. In particular, you can find most system configuration
details in easy-to-read plaintext files. The only tricky part is figuring out
which parts are responsible for what and how they all fit together.

Who Should Read This Book?

Your interest in learning how Linux works may have come from any number
of sources. In the professional realm, operations and DevOps folks need to
know nearly everything that you’ll find in this book. Linux software archi-
tects and developers should also know this material in order to make the
best use of the operating system. Researchers and students, often running
their own Linux systems, will also find that this book provides useful expla-
nations for why things are set up the way they are.
Then there are the tinkerers—people who just love to play around with
their computers for fun, profit, or both. Want to know why certain things
work while others don’t? Want to know what happens if you move some-
thing around? You’re probably a tinkerer.
 Prerequisites
Although Linux is beloved by programmers, you don’t need to be a pro-
grammer to read this book; you need only basic computer-user knowledge.
That is, you should be able to bumble around a GUI (especially the installer
and settings interface for a Linux distribution) and know what files and
directories (folders) are. You should also be prepared to check additional
documentation on your system and on the web. The most important thing
you need is to be ready and willing to play around with your computer.

How to Read This Book

Building the requisite knowledge is a challenge in tackling any technical
subject. Explaining how software systems work can get really complicated.
Too much detail can bog down readers and make important concepts dif-
ficult to grasp (the human brain just can’t process so many new ideas at
once), but too little detail can leave readers in the dark and unprepared for
later material.
I’ve designed most chapters to tackle the most important material first:
the basic information that you’ll need in order to progress. In places, I’ve
simplified things in order to keep focus. As a chapter progresses, you’ll see
much more detail, especially in the last few sections. Do you need to know
those bits right away? In most cases, no; I note this where applicable. If your
eyes start to glaze over when you’re faced with a lot of extra details about
concepts you just learned, don’t hesitate to skip ahead to the next chapter
or take a break. The nitty-gritty will still be there waiting for you.

A Hands-on Approach
However you choose to proceed through this book, you should have a Linux
machine in front of you, preferably one that you’re confident abusing with
experiments. You might prefer to play around with a virtual installation—I
used VirtualBox to test much of the material in this book. You also should
have superuser (root) access, but try to use a regular user account most of
the time. You’ll mostly work at the command line, in a terminal window or
a remote session. If you haven’t worked much in that environment, no prob-
lem; Chapter 2 will bring you up to speed.
Commands in this book typically look like this:

$ ls /
[some output]

Enter the text in bold; the nonbolded text that follows is what the
machine spits back. The $ is the prompt for a regular user account. If you
see a # as a prompt, you need to be superuser. (More on that in Chapter 2.)

xxii Preface
How This Book Is Organized
I’ve grouped the book’s chapters into three basic parts. The first is intro-
ductory, giving a bird’s-eye view of the system and then offering hands-on
experience with some tools you’ll need for as long as you run Linux. Next,
you’ll explore each part of the system in more detail, from device manage-
ment to network configuration, following the general order in which the
system starts. Finally, you’ll get a tour of some pieces of a running system,
learn some essential skills, and get some insight into the tools that program-
mers use.
With the exception of Chapter 2, most of the early chapters heavily
involve the Linux kernel, but you’ll work your way into user space as the
book progresses. (If you don’t know what I’m talking about here, don’t
worry; I’ll explain in Chapter 1.)
The material is meant to be as distribution-agnostic as possible. Having
said this, it can be tedious to cover all variations in systems software, so I’ve
tried to cover the two major distribution families: Debian (including Ubuntu)
and RHEL/Fedora/CentOS. I’ve also focused on desktop and server installa-
tions. A significant amount of material carries over into embedded systems,
such as Android and OpenWRT, but it’s up to you to discover the differences
on those platforms.

What’s New in the Third Edition?

The second edition was published during a time of transition for Linux
systems. Several traditional components were in the process of being
replaced, which made handling some subjects tricky, because readers could
encounter a wide variety of configurations. Now, however, those new pieces
(systemd in particular) have near-universal adoption, so I’ve been able to
streamline a fair amount of the discussion.
I’ve retained the emphasis on the kernel’s role in a Linux system. This
material has proven popular, and you probably interact with the kernel
more than you realize.
I’ve added a new chapter introducing virtualization. Although Linux
has always been popular on virtual machines (such as cloud services), that
type of virtualization lies outside the scope of this book, because the way
the system operates on a virtual machine is almost the same as on “bare
metal” hardware. So, the discussion here primarily focuses on decipher-
ing the terminology you’ll encounter. However, containers have gained in
popularity since the second edition was published, and they also fit here,
because they basically consist of a bunch of Linux features like the ones
described throughout the rest of the book. Containers make heavy use of
cgroups, which also get new treatment in this third edition.
Other subjects (not necessarily related to containers) I happily expanded
on are the Logical Volume Manager, the journald logging system, and IPv6
in the networking material.

Preface xxiii
 Though I’ve added a significant amount of content, the book is still a
reasonable size. I want to provide the information you need to get on the
fast track, and that includes explaining certain details along the way that
can be hard to grasp, but I don’t want you to have to become a weightlifter
in order to pick up this book. Once you master the important subjects here,
you should have no trouble seeking out and understanding more details.
The first edition included historical information that I removed later to
improve focus. If you’re interested in Linux and how it relates to the history
of Unix, pick up Peter H. Salus’s The Daemon, the Gnu, and the Penguin (Reed
Media Services, 2008). It does a great job of explaining how the software we
use has evolved over time.

A Note on Terminology
The names of certain operating system elements have caused a fair amount
of debate historically—even the word Linux itself. Should it be “Linux,” or
should it be “GNU/Linux” to reflect that the operating system also contains
pieces from the GNU Project? Throughout this book, I’ve tried to use the
most common, least awkward names possible.

xxiv Preface
 1
THE BIG PICTURE

At first glance, a contemporary operating

system such as Linux is very complicated,
with a dizzying number of pieces simultane-
ously running and communicating. For exam-
ple, a web server can talk to a database server, which
could in turn use a shared library that many other pro-
grams use. How does all of this manage to work, and
how can you make sense of any of it?
The most effective way to understand how an operating system works
is through abstraction—a fancy way of saying that you can ignore most of
the details that make up a piece that you’re trying to understand, and con-
centrate instead on its basic purpose and operation. For example, when
you ride in a car, you normally don’t need to think about details such as the
mounting bolts that hold the motor inside the car or the people who build
and maintain the road upon which the car drives. All you really need to
know is what the car does (transports you somewhere else) and a few basics
about how to use it (how to operate the door and seat belt).
 This level of abstraction might work if you’re just a passenger in the car.
But if you also need to drive it, you have to dig deeper and break up your
abstraction into a few parts. You now expand your knowledge in three areas:
the car itself (such as its size and capabilities), how to operate the controls
(steering wheel, accelerator pedal, and so on), and the features of the road.
Abstraction can be a great help when you’re trying to find and fix prob-
lems. For example, let’s say you’re driving a car and the ride is rough. You
can quickly assess the three basic car-related abstractions just mentioned to
determine the source of the problem. It should be fairly easy to eliminate
the first two abstractions (your car or the way you’re driving) if neither is the
issue, so you can narrow the problem down to the road itself. You’ll probably
find that the road is bumpy. Now, if you want, you can dig deeper into your
abstraction of the road and find out why the road has deteriorated or, if the
road is new, why the construction workers did a lousy job.
Software developers use abstraction as a tool when building an oper-
ating system and its applications. There are many terms for an abstracted
subdivision in computer software—including subsystem, module, and package—
but we’ll use the term component in this chapter because it’s simple. When
building a software component, developers typically don’t think much about
the internal structure of other components, but they do consider other
components they can use (so that they don’t have to write any additional
unnecessary software) and how to use them.
This chapter provides a high-level overview of the components that make
up a Linux system. Although each one has a tremendous number of technical
details in its internal makeup, we’re going to ignore these details and concen-
trate on what the components do in relation to the whole system. We’ll look
at the details in subsequent chapters.

1.1 Levels and Layers of Abstraction in a Linux System

Using abstraction to split computing systems into components makes things
easier to understand, but it doesn’t work without organization. We arrange
components into layers or levels, classifications (or groupings) of compo-
nents according to where the components sit between the user and the
hardware. Web browsers, games, and such sit at the top layer; at the bottom
layer we have the memory in the computer hardware—the 0s and 1s. The
operating system occupies many of the layers in between.
A Linux system has three main levels. Figure 1-1 shows these levels
and some of the components inside each level. The hardware is at the base.
Hardware includes the memory as well as one or more central processing
units (CPUs) to perform computation and to read from and write to memory.
Devices such as disks and network interfaces are also part of the hardware.
The next level up is the kernel, which is the core of the operating system.
The kernel is software residing in memory that tells the CPU where to look
for its next task. Acting as a mediator, the kernel manages the hardware
(especially main memory) and is the primary interface between the hard-
ware and any running program.

2 Chapter 1
 Processes—the running programs that the kernel manages—collectively
make up the system’s upper level, called user space. (A more specific term
for process is user process, regardless of whether a user directly interacts with
the process. For example, all web servers run as user processes.)

User Processes

Graphical User Interface Servers Shell

Linux Kernel

System Calls Process Management Memory Management

Device Drivers

Hardware

Processor (CPU) Main Memory (RAM) Disks Network Ports

Figure 1-1: General Linux system organization

There is a critical difference between how the kernel and the user pro-
cesses run: the kernel runs in kernel mode, and the user processes run in
user mode. Code running in kernel mode has unrestricted access to the pro-
cessor and main memory. This is a powerful but dangerous privilege that
allows the kernel to easily corrupt and crash the entire system. The memory
area that only the kernel can access is called kernel space.
User mode, in comparison, restricts access to a (usually quite small) sub-
set of memory and safe CPU operations. User space refers to the parts of main
memory that the user processes can access. If a process makes a mistake and
crashes, the consequences are limited and can be cleaned up by the kernel.
This means that if your web browser crashes, it probably won’t take down the
scientific computation that has been running in the background for days.
In theory, a user process gone haywire can’t cause serious damage to the
rest of the system. In reality, it depends on what you consider “serious dam-
age,” as well as the particular privileges of the process, because some pro-
cesses are allowed to do more than others. For example, can a user process
completely wreck the data on a disk? With the correct permissions, yes—and
you might consider this to be fairly dangerous. There are safeguards to pre-
vent this, however, and most processes simply aren’t allowed to wreak havoc
in this manner.

The Big Picture 3

 NOTE The Linux kernel can run kernel threads, which look much like processes but have
access to kernel space. Some examples are kthreadd and kblockd.

1.2 Hardware: Understanding Main Memory

Of all of the hardware on a computer system, main memory is perhaps the
most important. In its rawest form, main memory is just a big storage area
for a bunch of 0s and 1s. Each slot for a 0 or 1 is called a bit. This is where the
running kernel and processes reside—they’re just big collections of bits. All
input and output from peripheral devices flows through main memory, also
as a bunch of bits. A CPU is just an operator on memory; it reads its instruc-
tions and data from the memory and writes data back out to the memory.
You’ll often hear the term state in reference to memory, processes, the
kernel, and other parts of a computer system. Strictly speaking, a state is a
particular arrangement of bits. For example, if you have four bits in your
memory, 0110, 0001, and 1011 represent three different states.
When you consider that a single process can easily consist of millions
of bits in memory, it’s often easier to use abstract terms when talking about
states. Instead of describing a state using bits, you describe what something
has done or is doing at the moment. For example, you might say, “The pro-
cess is waiting for input” or, “The process is performing Stage 2 of its startup.”

NOTE Because it’s common to refer to the state in abstract terms rather than to the actual
bits, the term image refers to a particular physical arrangement of bits.

1.3 The Kernel

Why are we talking about main memory and states? Nearly everything that
the kernel does revolves around main memory. One of the kernel’s tasks is to
split memory into many subdivisions, and it must maintain certain state infor-
mation about those subdivisions at all times. Each process gets its own share
of memory, and the kernel must ensure that each process keeps to its share.
The kernel is in charge of managing tasks in four general system areas:
Processes  The kernel is responsible for determining which processes
are allowed to use the CPU.
Memory  The kernel needs to keep track of all memory—what is cur-
rently allocated to a particular process, what might be shared between
processes, and what is free.
Device drivers   The kernel acts as an interface between hardware
(such as a disk) and processes. It’s usually the kernel’s job to operate
the hardware.
System calls and support   Processes normally use system calls to com-
municate with the kernel.
We’ll now briefly explore each of these areas.

4 Chapter 1
NOTE If you’re interested in the detailed workings of a kernel, two good textbooks are
Operating System Concepts, 10th edition, by Abraham Silberschatz, Peter B.
Galvin, and Greg Gagne (Wiley, 2018), and Modern Operating Systems, 4th
edition, by Andrew S. Tanenbaum and Herbert Bos (Prentice Hall, 2014).

1.3.1 Process Management

Process management describes the starting, pausing, resuming, scheduling,
and terminating of processes. The concepts behind starting and terminat-
ing processes are fairly straightforward, but describing how a process uses
the CPU in its normal course of operation is a bit more complex.
On any modern operating system, many processes run “simultane-
ously.” For example, you might have a web browser and a spreadsheet open
on a desktop computer at the same time. However, things are not as they
appear: the processes behind these applications typically do not run at
exactly the same time.
Consider a system with a one-core CPU. Many processes may be able to
use the CPU, but only one process can actually use the CPU at any given
time. In practice, each process uses the CPU for a small fraction of a second,
then pauses; then another process uses the CPU for another small fraction
of a second; then another process takes a turn, and so on. The act of one pro-
cess giving up control of the CPU to another process is called a context switch.
Each piece of time—called a time slice—gives a process enough time for
significant computation (and indeed, a process often finishes its current
task during a single slice). However, because the slices are so small, humans
can’t perceive them, and the system appears to be running multiple pro-
cesses at the same time (a capability known as multitasking).
The kernel is responsible for context switching. To understand how this
works, let’s think about a situation in which a process is running in user
mode but its time slice is up. Here’s what happens:

1. The CPU (the actual hardware) interrupts the current process based on
an internal timer, switches into kernel mode, and hands control back to
the kernel.
2. The kernel records the current state of the CPU and memory, which
will be essential to resuming the process that was just interrupted.
3. The kernel performs any tasks that might have come up during the
preceding time slice (such as collecting data from input and output, or
I/O, operations).
4. The kernel is now ready to let another process run. The kernel analyzes
the list of processes that are ready to run and chooses one.
5. The kernel prepares the memory for this new process and then pre-
pares the CPU.
6. The kernel tells the CPU how long the time slice for the new process
will last.
7. The kernel switches the CPU into user mode and hands control of the
CPU to the process.

The Big Picture 5

Another Random Scribd Document
with Unrelated Content
Tan Shoes, To Keep from Turning Dark, 377
Tandem, Three-Wheel, Bicycles Driven as, After Breakdown, 410
Tangling of Telephone Cord, Rubber Band Prevents, 367
Tangling, Preventing Sewing-Machine Thread from, 382
Tank, Miniature Fighting, 202
Tank, Reversible Photographic Developing, 325
Target Box, Illuminated Indicating, 180
Target, Recording Annunciator, 310
Teakettle Cover Held by Dent in Edge, 5
Team, Support for Wagon Pole Aids in Hooking Up, 5
Teaspoon, Catching Large Fish with, 401
Tee, Golf, Made of a Shotgun Shell, 430
Telegraph Recorder with Spool-and-Pencil Indicator, 171
Telegraph, Signal, with Green and Red Lights, 176
Telegraph Sounder, Battery Buzzer Converted into, 141
Telephone Cord, Rubber Band Prevents Tangling of, 367
Telescope, Interesting Water, 228
Temperature of a Room, Raising, 356
Temperature, Proper, Heater Keeps Developer at, 190
Temperature Readings, Transposing, 376
Temporary Binder for Magazines, 370
Tender, Bicycle Carried on Automobile, 190
Tennis-Court Backstop, Knock-Down, 189
Tennis-Court Marker, Wheelbarrow as, 397
Tennis Court, Removable Posts for, 415
Tennis Court, Scraper for, 311
Tennis Courts, To Keep Grass and Weeds Out of, 149
Tennis Nets, Tightening Lever for, 158
Tennis-Racket Clamp, Opening Springs for, 393
Tenpins, Home, Pin Setter for, 61
Tension Weight, Emergency, Used on Typewriter, 457
Tent, Hammock Sleeping, 242
Tent, Homemade Shoulder-Pack, 131
Tent, Portable, Made from an Umbrella and Paper Muslin, 364
Test Exposures for Bromide Enlargements, 411
Test-Tube Flower Vase, Stand for, 21
Test Tube, Mucilage Brush and Container Made of, 335
Testing and Caring for Files, 400
Testing Direct Current Polarity with Litmus Paper, 369
Testing Dry Cells with Light Bulb, 267
Theft, Guarding Camp Chest Against, 362
Thermometer, Joining Broken Liquid Column, 366
Thread and Dental Floss, Sanitary Holder for, 46
Thread Fabric, Electric Counting Glass for, 321
Thread, Sewing-Machine, Preventing from Tangling, 382
Threader, Needle, for Sewing Machine, 134
Threading a Darning Needle, 153
Three-Caster Truck for Moving Crates and Furniture, 419
Thresher, Small, of Practical Use, 444
Ticket Holder, Revolving Card or, 369
Ticking of a Watch or Clock, Muffling, 223
Tightening Lever for Tennis Nets, 158
Tightening Wire Spokes in Wheels, Handy Tool for, 450
Tile Trap for Rabbits, 184
Tilting Top for Camera Tripod, 242
Time, Blotter Attached to Wrist Saves, 295
Timing Pointer on Watch Crystal, 364
Tin Can, Camp Lantern Made of, 406
Tin Can, Case for Fishhooks Made of, 267
Tin Can, Cheese Grater and Ash Tray Made from, 225
Tin Can on Rod for Picking Fruit, 54
Tin Cover, Frying Pan Made of, 298
Tin Pail, Safety Flue Stopper, 328
Tinned Staples for Bell-Circuit Wiring, 420
Tinware, Cleaning with Milk, 44
Tire, Motor-Car, Changing Without a Jack, 210
Tire Patch, 400
Tire Pump, Detachable Motor-Driven, Made of Foot Pump, 454
Tire Pump Made of Gas Piping, 419
Tire Tubes, Checking a Rip in, 354
Tires, Baby-Cab, Changing Wheels Equalizes Wear on, 446
Tires, Economy in Motorcycle, 188
Toboggan and Slide, Water-Coasting, 183
Toboggan, Coasting, Making a, 11
Toboggan Made of Old Buggy Shafts, 410
Toboggan, Rudder for, 323
Toe Clips, Homemade Snowshoe, 418
Toilet Articles, Sundries and, Soldier’s or Traveler’s Kit for, 452
Toilet Outfit, Compact, for the Soldier, 9
Tool-Chest Slide Tray, Nonbinding, 371
Tool Drawer, Convenient, Under Chair Seat, 169
Tool Handles, Ferrules for, 157
Tool, Handy, for Tightening Wire Spokes in Wheels, 450
Tool, Oiling, for Clocks, 107
Tool Rack, Yardstick on, 417
Tool Sockets in Edge of Drawing Board, 435
Tool, Weeding, 248
Tools, Keeping Bright and Free from Rust, 212
Toothbrush, Uses for Old, 428
Tooth Powder Used on Tracing Cloth, 346
Top of Drawer, Desk Slide in, 356
Top, Split-Bamboo Tray for, Folding Table with, 424
Top, Tilting, for Camera Tripod, 242
Tossing Card at Mark Accurately, 69
Towel, Disappearing, 154
Tower, Diving, for the Summer Camp, 274
Toy Alligator of Wood, Mechanical, 460
Toy Electric Motor, Quickly Made, 252
Toy Horse That Walks, 363
Toy Machine Gun Fires Wooden Bullets, 408
Toy Paper Glider Carefully Designed, 324
Toy Paper Warships, 293
Toy Pigeon, Mechanical, Made of Wood, 433
Toy Rubber Balloons, Filling with Hydrogen, 30
Toy Submarine Made of Shade Roller, 441
Toy Tractor Built with Dry Cell and Motor, 9
Toy, Wire-Walking, 180
Toys and Boxes Made at Home, Decorative, 299
Tracing-Cloth Drawings, Mounting on Muslin, 418
Tracing Cloth, Tooth Powder Used on, 346
Track System, Homemade Electric Locomotive Model and:
 Part I.—The Motor, 231
Part II.—Construction of the Locomotive Truck and Cab, 237
Part III.—Construction of the Track System, 243
Tractor, Toy, Built with Dry Cell and Motor, 9
Train Berth, Improvised Trousers Hanger in, 367
Train, Writing on Moving, 228
Trammels, How to Make a Pair of, 166
Transfer Drawings, How to, 138
Transferring Pictures to Glass, 443
Transmitting Writing, Simple Machine for, 442
Transplanting, Device for Packing Earth in, 211
Transposing Temperature Readings, 376
Trap for Coyotes, 306
Trap for Rabbits, Tile, 184
Trap Nest for the Poultry House, 455
Trap, Self-Setting Rat, 31
Trapdoor, Floor, Fuel Box in Seat Filled from, 332
Traveler’s Kit, Soldier’s or, for Sundries and Toilet Articles, 453
Tray Attachment for Developing Films, 280
Tray, Developing, Cooler for, 149
Tray for Top, Split-Bamboo, Folding Table with, 424
Tray, Handy, for Pencils and Penholders, 430
Tray, Liquid-Filled, Carried Safely, 439
Tray, Nonbinding Tool-Chest Slide, 371
Trays, Cocoanut-Shell, 414
Trays, Index, Novel Covered Box for, 414
Trays, Making Photographic, 406
Trays, Muffin-Pan, Nail Cabinet with, 230
Tree Stump, Ornamenting Old, 123
Tree Stumps, Lawn Seats Built on, 141
Trellis, Rustic, to Shade Door or Window, 175
Trenches, Miniature Fighting Tank That Hurdles, 202
Trick Blotter, 354
Trick, Diminishing Card, 396
Trick, Disappearing-Coin, 144
Trick, Finger-Trap, 377
Trick, Glass-and-Hat, 342
Trick, Match-Box, 230
Trick, Ring-and-Egg, 84
Trick, Vanishing-Cuff Parlor, 127
Tricks of Camping Out:
Part I.—The Camping Outfit, 109
Part II.—Cooking in the Woods, 117
Trim Magazines for Binding, Knife to, 286
Trimmer, Photo and Paper, Homemade Guide for, 366
Trimming Board with Foot Control and Counterweight, 308
Trinket Case for the Bookshelf, Secret, 296
Tripod, Camera, Tilting Top for, 242
Trips, Memorandum List for Camping and Outing, 365
Trophy Cup, Onlaying Script on, 188
Trouble Lamp, Homemade, 365
Trouser Hanger, 350
Trousers Hanger, Improvised, in Train Berth, 367
Truck, Gravity-Feed Coal Hopper on, 140
Truck, Three-Caster, for Moving Crates and Furniture, 419
Trunk Bookcase for Convenient Shipment, 217
Tub, Combination Laundry and Dishwashing Sink, 218
Tuberculosis Cottage, A, 385
Tubes, Cardboard, for Electrical Coils, Making, 438
Tubes, Old Inner, Rubber Bands Made from, 268
Tubes, Paste, Wall Pocket for, 16
Tubes, Squeezing Paste from, 391
Tubing, Cane Made of, Contains Cigars, 430
Tug-of-War Game, Device for Finger, 319
Turbine, Small Hydraulic, 427
Turned Cane with Snakes Inlaid, 325
Turning Long Wood Rods, 349
Turning Sheet Music Quickly, Tabs for, 368
Turning, To Prevent Wire Coat Hook from, 235
Turntable Stand for Potted Flowers, 308
Twine Hammock, How to Make, 277
Twisting Thriller Merry-Go-Round, 179
Two Colors, Writing, on Plain-Ribbon Typewriter, 168
Type Cases, Cleaning with Bicycle Pump, 451
Typewriter Desk, Improvised, 225
Typewriter, Emergency Tension Weight Used on, 457
Typewriter, Feeding Cards into, 437
Typewriter, Kinks in Cleaning, 6
Typewriter, Plain-Ribbon, Writing Two Colors on, 168
Typewriter, Roll-Paper Feed for, 207
Typewriter Space and Shift Keys, Pedals for Increase Speed, 364
Typewriter, Wire Clips Weight Paper in, 409
Typewritten Bound Sheets, Inserting or Correcting on, 419
Ukulele, Homemade Hawaiian, 358
Umbrella and Paper Muslin, Portable Tent Made from, 364
Umbrella Handle, Making Detachable, 439
Umbrella Used as a Clothes Drier, 366
Umbrellas, Kinks on Care of, 422
Uncork a Bottle with a String, To, 402
Unsealed Envelopes, Safeguarding Contents of, 363
Use for Old Magazines, 399
Uses for an Old Toothbrush, 428
Uses for Worn Talking-Machine Needles, 329
Utensil Rack for Camp Fire, 397
Utensils, Kitchen, Cupboard for, 396
Utilizing an Empty Paste Pot, 306
Vacuum Pail, How to Make, 315
Valve-Bottom Pail for Dipping Water, 344
Valve, Radiator, Ship’s-Wheel Device for, 259
Valves on Gas Stove, Safety Cover for, 298
Vanishing-Cuff Parlor Trick, 127
Vaporizer, Homemade, 346
Variable Condenser, Small, 334
Vase, Test-Tube Flower, Stand for, 21
Vaulting Pole, Scale on, Indicates Points of Grip, 411
Vehicle, Child’s Play, Auto Horn for, 16
Ventilation, Thorough, Bedroom Shade and Curtains Arranged for,
128
Ventilator, Window, 312
Veranda, Summer, Taborets and Small Tables for, 269
Vest, Guard Saves Wear on, 413
Vibrator, Double-Contact, 140
View Finder, Camera, Rectangular Opening to Use Over, 125
View Finder, Direct, for Box Camera, 353
Views, Fireside Dissolving, 351
Vise, Jig-Saw Table for, 93
Vise, Quick-Acting Bench, 85
Vises for the Home Workbench, Two Simple, 197
Vulcanizer, Homemade, 323
Wagon-Wheel Felly and Spoke, Mallet Made from, 157
Wagon Pole, Support for, Aids in Hooking Up Team, 5
Wagon Seat, Homemade Spring, 440
Wall Back of Range or Sink, Protecting, 354
Wall Desk, Folding, 292
Wall Fastening, Ironing-Board, 318
Wall Paper, Kink for Removal of, 295
Wall, Plaster of Paris to Set Screws into, 266
Wall Pocket for Paste Tubes, 16
Wall Shelves, Easily Constructed, 108
Wall Workbench, Old Table Used as, 440
Walls, Damp, Shielding Pictures from, 338
Walks, Toy Horse That, 363
Warships, Toy Paper, 293
Washing Bromide Enlargements, 336
Washing Device, Automatic Photo-Print, 329
Washing Machine Equipped with Churn Attachment, 208
Washing Machine, Photographic-Print, 327
Washing Photographic Negatives and Prints, Kinks in, 181
Washstand for the Baby, 328
Wastebasket, False Bottom for Emptying, 344
Watch Bezel, Screw, Rubber Pads for Opening, 448
Watch Crystal, Timing Pointer on, 364
Watch Holder, Desk, 158
Watch, Mysterious, 70
Watch or Clock, Muffling the Ticking of, 223
Water Bag Camp, 122
Water Basin for Poultry, Concrete, 236
Water-Coasting Toboggan and Slide, 183
Water Heater, Carbon Electric, 356
Water, Irrigation, Current of Canal Raises, 411
Water-Jacket Outlet, Bilge Water Siphoned Through, 413
Water Pipe, Broken Spade Handle Repaired with, 242
Water Rheostat for Small Electrical Devices, 196
Water Telescope, Interesting, 228
Water, Valve-Bottom Pail for Dipping, 344
Water Wheel Turns Spit Over Campfire, 429
Watering Window-Box Flowers, 144
Waterproof Dry-Battery Case, 265
Waterproofing for Fish Lines, 94
Waterproofing Matches, 230
Wear on Baby-Cab Tires, Changing Wheels Equalizes, 446
Wear on Drafting Board, Cork Plugs Save, 21
Wear on Vest, Guard Saves, 413
Webfoot Attachments for Swimmers, 381
Weeding Garden, Forceps for, 338
Weeding Tool, 248
Weeds, Grass and, To Keep Out of Tennis Courts, 149
Weight, Emergency Tension, Used on Typewriter, 457
Weighted Rope Holds Flag Upright, 451
Weighting a Metal Base, 217
Well, Rustic, for Bazaar or Fair Booth, 182
Wheel, China Banding, Disk Talking Machine as, 10
Wheelbarrow as Tennis-Court Marker, 397
Wheelbarrow for Large Cans, 330
Wheels, Changing, Equalizes Wear on Baby-Cab Tires, 446
Wheels, Handy Tool for Tightening Wire Spokes in, 450
Wheels, Model Gear, Gauge for Laying Out, 384
Whipping of Flag, Chain Weight Prevents, 409
Whirligig, Perpetual, 400
Whirling Fan, Hand-Operated, 398
Whirling Fan, Lighted, Used as Radiator Ornament, 260
Whistle, Hand-Operated Motorboat, 178
Whistle Warns of Fish Catch, 275
White Blotting Paper Improves Light Reflectors, 196
Wick, Feeding, Stove Lighter with, Guards Against Burns, 459
Wick, Lighting Candle Without Touching, 334
Wicks, Lamp, Cheaply Made, 236
Wind, Lighting a Match in, 382
Wind Motor, Sail Rigged, 172
Wind Wire on Electrical Apparatus, How to, 136
Winder, Egg Beater Made into for Model Aeroplanes, 459
Winding Coiled Springs, 134
Window-Advertising Novelty, Moth-Ball Puzzle as, 444
Window-Box Flowers, Watering, 144
Window Box, Hinged, 413
Window Closer, Automatic, 280
Window Display, Revolving, 229
Window Frame and Table for Dark Room, 320
Window Refrigerator, 323
Window, Rustic Trellis to Shade Door or, 175
Window Sash, Locking, 62
Window Ventilator, 312
Windows, Horizontal Sliding, Hinge Lock for, 372
Windows, To Stop Rattling of, 417
Wings, Covering Hinge, 276
Winter, Making Use of Refrigerator in, 344
Winter, Summer Radiator Cover Serves as Cupboard in, 297
Winter Use, Bee Feeder for, 192
Winter Use, Bicycle Runner for, 418
Wire Clips Weight Paper in Typewriter, 409
Wire Coat Hook, To Prevent from Turning, 235
Wire Compacts Bristles in Polish or Stencil Brushes, 439
Wire Holders Keep Cabinet Doors Open, 127
Wire, How to Wind on Electrical Apparatus, 136
Wire-Mesh Cage, Fisherman’s Pail with, 454
Wire Mesh, Preventing from Rising Between Fence Posts, 93
Wire-Mesh Support for Flower Centerpiece, 344
Wire Netting, Asbestos Table Mats Reinforced with, 421
Wire-Screen Pincushion, 456
Wire Spokes in Wheels, Handy Tool for Tightening, 450
Wire Trellis Fastened Neatly to Brick Walls, 8
Wire-Walking Toy, 180
Wireless Aerials, Lightning Switch for, 415
Wireless Detector, Simple, 456
Wiring, Bell-Circuit, Tinned Staples for, 420
Wishbone-Mast Ice Yacht, 17
Wood Alcohol, Economical Use of in Small Cooking Stove, 210
Wood Box with a Refuse-Catching Drawer, 144
Wood, Driving Thin Metal into, 247
Wood for Cabinetwork, Storage of, 389
Wood, Groove Cutter for, 45
Wood, Hard, Driving Screws in, 94
Wood, Mechanical Toy Alligator of, 460
Wood, Mechanical Toy Pigeon Made of, 433
Wood Rods, Turning Long, 349
Wood Turning on an Emery Grinder, 402
Wood-Wind Instruments, Repairing, 174
Wood, Working by Application of Heat, 150
Wooden Bullet, Toy Machine Gun Fires, 408
Wooden Disks, Cutting Thin, 16
Wooden Strips, Enameled Armchair Made of, 129
Woods, Birch-Bark Leggings Made of, 421
Woods, Cooking in the, 117
Woodsman’s Log Raft, 185
Woodwork, Gauge for, 252
Workbag in Top, Sewing Stand with, 293
Workbench, Home, Two Simple Vises for, 197
Workbench, Pencil Holder for, 236
Workbench, Placing Miter Box on, 294
Workbench, Wall, Old Table Used as, 440
Working Pile Driver, Small, 215
Working Wood by Application of Heat, 150
Workshop Seat, Combination, 370
Worn Talking-Machine Needles, Uses for, 329
Woven-Reed Footstool, 255
Woven Reed Furniture, 261, 269
Wrap Papers, Proper Way for Mailing, 44
Wrench, Fountain-Pen, 273
Wrist, Blotter Attached to Saves Time, 295
Writing and Drawing Pad, Cardboard, 130
Writing Desk, Combination Bookcase and, 316
Writing, Homemade Device Aids Blind Person, 438
Writing on a Moving Train, 228
Writing, Simple Machine for Transmitting, 442
X-Ray Lens, Feather as, 412
Yardstick on Tool Rack, 417
 Transcriber’s Notes

The language used in this text is that of the sourcce

document; changes to the text are listed below.
Depending on the hard- and software and their
settings used to read this text not all elements
may display as intended; many images may be
enlarged by opening them in a new window or tab.
Page 32, ... an angle of about 2°: possibly an error
for ... an angle of about 20° (which is the angle in
the illustration).
Page 113, illustration top right (Food Bags): Fut and
Ergswurst may be errors for Fat and Erbswurst
(pea sausage).
Page 262, ... in Fig. 9, illustrating an article on
“Taborets and Small Tables for the Summer
Veranda,” page 155, July, 1916 ...: an article with
this title (presumably the one referred to) is
present in this book on page 269.
Changes made
Illustrations have been moved out of text
paragraphs.
Minor obvious typographical errors have been
corrected silently.
Text in a dotted box has been transcribed from the
accompanying illustration, and does not appear as
part of the text in the source document.
Some entries in the table of contents have been
corrected to conform to the spelling used in the
text.
In some of the illustrations feet or misprinted inch
symbols (′) have been changed to inch symbols (″)
when necessary.
On several pages the first letter of the main article
has been replaced with a large capital, as in
similar articles in the book.
Page 29: Fig 7 has been rotated 90°.
Page 89: ... directly from the canoe, or part, to be
fitted, whenever convenient ... changed to ...
directly from the canoe, or part to be fitted,
whenever convenient ....
Page 94: illustration Ice Creeper turned upside-
down.
Page 213: ... a wear and noise-proof bearing ...
changed to ... a wear- and noise-proof bearing ....
*** END OF THE PROJECT GUTENBERG EBOOK THE BOY
MECHANIC, BOOK 3 ***

Updated editions will replace the previous one—the old editions

will be renamed.

Creating the works from print editions not protected by U.S.

copyright law means that no one owns a United States copyright
in these works, so the Foundation (and you!) can copy and
distribute it in the United States without permission and without
paying copyright royalties. Special rules, set forth in the General
Terms of Use part of this license, apply to copying and
distributing Project Gutenberg™ electronic works to protect the
PROJECT GUTENBERG™ concept and trademark. Project
Gutenberg is a registered trademark, and may not be used if
you charge for an eBook, except by following the terms of the
trademark license, including paying royalties for use of the
Project Gutenberg trademark. If you do not charge anything for
copies of this eBook, complying with the trademark license is
very easy. You may use this eBook for nearly any purpose such
as creation of derivative works, reports, performances and
research. Project Gutenberg eBooks may be modified and
printed and given away—you may do practically ANYTHING in
the United States with eBooks not protected by U.S. copyright
law. Redistribution is subject to the trademark license, especially
commercial redistribution.

START: FULL LICENSE

THE FULL PROJECT GUTENBERG LICENSE
 PLEASE READ THIS BEFORE YOU DISTRIBUTE OR USE THIS WORK

To protect the Project Gutenberg™ mission of promoting the

free distribution of electronic works, by using or distributing this
work (or any other work associated in any way with the phrase
“Project Gutenberg”), you agree to comply with all the terms of
the Full Project Gutenberg™ License available with this file or
online at www.gutenberg.org/license.

Section 1. General Terms of Use and

Redistributing Project Gutenberg™
electronic works
1.A. By reading or using any part of this Project Gutenberg™
electronic work, you indicate that you have read, understand,
agree to and accept all the terms of this license and intellectual
property (trademark/copyright) agreement. If you do not agree to
abide by all the terms of this agreement, you must cease using
and return or destroy all copies of Project Gutenberg™
electronic works in your possession. If you paid a fee for
obtaining a copy of or access to a Project Gutenberg™
electronic work and you do not agree to be bound by the terms
of this agreement, you may obtain a refund from the person or
entity to whom you paid the fee as set forth in paragraph 1.E.8.

1.B. “Project Gutenberg” is a registered trademark. It may only

be used on or associated in any way with an electronic work by
people who agree to be bound by the terms of this agreement.
There are a few things that you can do with most Project
Gutenberg™ electronic works even without complying with the
full terms of this agreement. See paragraph 1.C below. There
are a lot of things you can do with Project Gutenberg™
electronic works if you follow the terms of this agreement and
help preserve free future access to Project Gutenberg™
electronic works. See paragraph 1.E below.
1.C. The Project Gutenberg Literary Archive Foundation (“the
Foundation” or PGLAF), owns a compilation copyright in the
collection of Project Gutenberg™ electronic works. Nearly all the
individual works in the collection are in the public domain in the
United States. If an individual work is unprotected by copyright
law in the United States and you are located in the United
States, we do not claim a right to prevent you from copying,
distributing, performing, displaying or creating derivative works
based on the work as long as all references to Project
Gutenberg are removed. Of course, we hope that you will
support the Project Gutenberg™ mission of promoting free
access to electronic works by freely sharing Project
Gutenberg™ works in compliance with the terms of this
agreement for keeping the Project Gutenberg™ name
associated with the work. You can easily comply with the terms
of this agreement by keeping this work in the same format with
its attached full Project Gutenberg™ License when you share it
without charge with others.

1.D. The copyright laws of the place where you are located also
govern what you can do with this work. Copyright laws in most
countries are in a constant state of change. If you are outside
the United States, check the laws of your country in addition to
the terms of this agreement before downloading, copying,
displaying, performing, distributing or creating derivative works
based on this work or any other Project Gutenberg™ work. The
Foundation makes no representations concerning the copyright
status of any work in any country other than the United States.

1.E. Unless you have removed all references to Project

Gutenberg:

1.E.1. The following sentence, with active links to, or other

immediate access to, the full Project Gutenberg™ License must
appear prominently whenever any copy of a Project
Gutenberg™ work (any work on which the phrase “Project
Gutenberg” appears, or with which the phrase “Project
Gutenberg” is associated) is accessed, displayed, performed,
viewed, copied or distributed:

This eBook is for the use of anyone anywhere in the United

States and most other parts of the world at no cost and with
almost no restrictions whatsoever. You may copy it, give it
away or re-use it under the terms of the Project Gutenberg
License included with this eBook or online at
www.gutenberg.org. If you are not located in the United
States, you will have to check the laws of the country where
you are located before using this eBook.

1.E.2. If an individual Project Gutenberg™ electronic work is

derived from texts not protected by U.S. copyright law (does not
contain a notice indicating that it is posted with permission of the
copyright holder), the work can be copied and distributed to
anyone in the United States without paying any fees or charges.
If you are redistributing or providing access to a work with the
phrase “Project Gutenberg” associated with or appearing on the
work, you must comply either with the requirements of
paragraphs 1.E.1 through 1.E.7 or obtain permission for the use
of the work and the Project Gutenberg™ trademark as set forth
in paragraphs 1.E.8 or 1.E.9.

1.E.3. If an individual Project Gutenberg™ electronic work is

posted with the permission of the copyright holder, your use and
distribution must comply with both paragraphs 1.E.1 through
1.E.7 and any additional terms imposed by the copyright holder.
Additional terms will be linked to the Project Gutenberg™
License for all works posted with the permission of the copyright
holder found at the beginning of this work.

1.E.4. Do not unlink or detach or remove the full Project

Gutenberg™ License terms from this work, or any files
containing a part of this work or any other work associated with
Project Gutenberg™.
 1.E.5. Do not copy, display, perform, distribute or redistribute
this electronic work, or any part of this electronic work, without
prominently displaying the sentence set forth in paragraph 1.E.1
with active links or immediate access to the full terms of the
Project Gutenberg™ License.

1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form,
including any word processing or hypertext form. However, if
you provide access to or distribute copies of a Project
Gutenberg™ work in a format other than “Plain Vanilla ASCII” or
other format used in the official version posted on the official
Project Gutenberg™ website (www.gutenberg.org), you must, at
no additional cost, fee or expense to the user, provide a copy, a
means of exporting a copy, or a means of obtaining a copy upon
request, of the work in its original “Plain Vanilla ASCII” or other
form. Any alternate format must include the full Project
Gutenberg™ License as specified in paragraph 1.E.1.

1.E.7. Do not charge a fee for access to, viewing, displaying,

performing, copying or distributing any Project Gutenberg™
works unless you comply with paragraph 1.E.8 or 1.E.9.

1.E.8. You may charge a reasonable fee for copies of or

providing access to or distributing Project Gutenberg™
electronic works provided that:

• You pay a royalty fee of 20% of the gross profits you derive from
the use of Project Gutenberg™ works calculated using the
method you already use to calculate your applicable taxes. The
fee is owed to the owner of the Project Gutenberg™ trademark,
but he has agreed to donate royalties under this paragraph to
the Project Gutenberg Literary Archive Foundation. Royalty
payments must be paid within 60 days following each date on
which you prepare (or are legally required to prepare) your
periodic tax returns. Royalty payments should be clearly marked
as such and sent to the Project Gutenberg Literary Archive
Foundation at the address specified in Section 4, “Information
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com