Patterns Of Distributed Systems 1st Edition

Unmesh Joshi download

https://ebookbell.com/product/patterns-of-distributed-
systems-1st-edition-unmesh-joshi-55578354

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Patterns Of Distributed Systems Unmesh Joshi

https://ebookbell.com/product/patterns-of-distributed-systems-unmesh-
joshi-48497112

Agricultural Specialization And Rural Patterns Of Development Rural

History In Europe Annie Antoine Editor

https://ebookbell.com/product/agricultural-specialization-and-rural-
patterns-of-development-rural-history-in-europe-annie-antoine-
editor-50329396

Remoting Patterns Foundations Of Enterprise Internet And Realtime

Distributed Object Middleware Markus Volter Michael Kircher Uwe Zdun

https://ebookbell.com/product/remoting-patterns-foundations-of-
enterprise-internet-and-realtime-distributed-object-middleware-markus-
volter-michael-kircher-uwe-zdun-4701206

Edge Computing Patterns For Solution Architects Learn Methods And

Principles Of Resilient Distributed Application Architectures From
Hybrid Cloud To Far Edge Ashok Iyengar Joseph Pearson

https://ebookbell.com/product/edge-computing-patterns-for-solution-
architects-learn-methods-and-principles-of-resilient-distributed-
application-architectures-from-hybrid-cloud-to-far-edge-ashok-iyengar-
joseph-pearson-58637174
Patterns Of World History Combined Volume With Sources Sample 2nd
Revised Ed Associate Professor Of History Peter Von Sivers

https://ebookbell.com/product/patterns-of-world-history-combined-
volume-with-sources-sample-2nd-revised-ed-associate-professor-of-
history-peter-von-sivers-44888366

Patterns Of Parental Behavior From Animal Science To Comparative

Ethology And Neuroscience Gabriela Gonzlezmariscal

https://ebookbell.com/product/patterns-of-parental-behavior-from-
animal-science-to-comparative-ethology-and-neuroscience-gabriela-
gonzlezmariscal-46462790

Patterns Of Exposition 18th Edition 18th Edition Robert A Schwegler

https://ebookbell.com/product/patterns-of-exposition-18th-
edition-18th-edition-robert-a-schwegler-48674118

Patterns Of Interaction Computational Design Across Scales Pia Fricker

https://ebookbell.com/product/patterns-of-interaction-computational-
design-across-scales-pia-fricker-49114422

Patterns Of Christological Categorisation Oneness Pentecostalism And

The Renewal Of Jewish And Christian Monotheism Marvin C Sanguinetti

https://ebookbell.com/product/patterns-of-christological-
categorisation-oneness-pentecostalism-and-the-renewal-of-jewish-and-
christian-monotheism-marvin-c-sanguinetti-49459098
Patterns of
Distributed Systems
 Pearson Addison-Wesley
Signature Series

Visit informit.com/awss for a complete list of available publications.

T he Pearson Addison-Wesley Signature Series provides readers

with practical and authoritative information on the latest trends in
modern technology for computer professionals. The series is based on
one simple premise: great books come from great authors.

Books in the Martin Fowler Signature Series are personally chosen

by Fowler, and his signature ensures that he has worked closely with
authors to define topic coverage, book scope, critical content, and
overall uniqueness. The expert signatures also symbolize a promise to
our readers: you are reading a future classic.

Connect with InformIT—Visit informit.com/community

Patterns of
Distributed Systems

Unmesh Joshi

Hoboken, New Jersey

Cover image: Joe Ravi/Shutterstock
Additional image credits appear on page 427, which constitutes a continuation of this
copyright page.

Many of the designations used by manufacturers and sellers to distinguish their products
are claimed as trademarks. Where those designations appear in this book, and the publisher
was aware of a trademark claim, the designations have been printed with initial capital
letters or in all capitals.

The author and publisher have taken care in the preparation of this book, but make no
expressed or implied warranty of any kind and assume no responsibility for errors or
omissions. No liability is assumed for incidental or consequential damages in connection
with or arising out of the use of the information or programs contained herein.

For information about buying this title in bulk quantities, or for special sales opportunities
(which may include electronic versions; custom cover designs; and content particular to
your business, training goals, marketing focus, or branding interests), please contact our
corporate sales department at [email protected] or (800) 382-3419.

For government sales inquiries, please contact [email protected].

For questions about sales outside the U.S., please contact [email protected].

Visit us on the Web: informit.com/aw

Library of Congress Control Number: 2023944564

Copyright © 2024 Pearson Education, Inc.

All rights reserved. This publication is protected by copyright, and permission must be
obtained from the publisher prior to any prohibited reproduction, storage in a retrieval
system, or transmission in any form or by any means, electronic, mechanical, photocopying,
recording, or likewise. For information regarding permissions, request forms and the
appropriate contacts within the Pearson Education Global Rights & Permissions
Department, please visit www.pearson.com/permissions.

ISBN-13: 978-0-13-822198-0
ISBN-10: 0-13-822198-7

$PrintCode
Pearson’s Commitment to Diversity, Equity, and Inclusion
Pearson is dedicated to creating bias-free content that reflects the diversity of all
learners. We embrace the many dimensions of diversity, including but not limited
to race, ethnicity, gender, socioeconomic status, ability, age, sexual orientation,
and religious or political beliefs.
Education is a powerful force for equity and change in our world. It has the
potential to deliver opportunities that improve lives and enable economic mobil-
ity. As we work with authors to create content for every product and service, we
acknowledge our responsibility to demonstrate inclusivity and incorporate diverse
scholarship so that everyone can achieve their potential through learning. As
the world’s leading learning company, we have a duty to help drive change and
live up to our purpose to help more people create a better life for themselves
and to create a better world.
Our ambition is to purposefully contribute to a world where:

Everyone has an equitable and lifelong opportunity to succeed through

learning.
Our educational products and services are inclusive and represent the rich
diversity of learners.
Our educational content accurately reflects the histories and experiences of
the learners we serve.
Our educational content prompts deeper discussions with learners and
motivates them to expand their own learning (and worldview).

While we work hard to present unbiased content, we want to hear from you
about any concerns or needs with this Pearson product so that we can investigate
and address them.

Please contact us with concerns about any potential bias at

https://www.pearson.com/report-bias.html.
This page intentionally left blank
Dedicated to the loving memory of
my father.
This page intentionally left blank
Contents

Foreword ......................................................................................................... xvii

Preface ............................................................................................................. xix
Acknowledgments .......................................................................................... xxiii
About the Author ............................................................................................. xxv

Part I: Narratives ................................................................................. 1

Chapter 1: The Promise and Perils of Distributed Systems ............................ 3
The Limits of a Single Server ............................................................................ 3
Separate Business Logic and Data Layer .......................................................... 5
Partitioning Data ................................................................................................. 6
A Look at Failures ............................................................................................... 7
Replication: Masking Failures ............................................................................ 9
Process Crash ................................................................................................ 9
Network Delay .............................................................................................. 9
Process Pause ................................................................................................ 9
Unsynchronized Clocks ............................................................................. 10
Defining the Term “Distributed Systems” ...................................................... 10
The Patterns Approach ..................................................................................... 10

Chapter 2: Overview of the Patterns ................................................................ 13

Keeping Data Resilient on a Single Server .................................................... 14
Competing Updates .......................................................................................... 15
Dealing with the Leader Failing ...................................................................... 17
Multiple Failures Need a Generation Clock ................................................... 21
Log Entries Cannot Be Committed until They Are Accepted by a
Majority Quorum .............................................................................................. 26
Followers Commit Based on a High-Water Mark ......................................... 29

ix
x Contents

Leaders Use a Series of Queues to Remain Responsive to Many

Clients ................................................................................................................ 34
Followers Can Handle Read Requests to Reduce Load on the Leader ...... 40
A Large Amount of Data Can Be Partitioned over Multiple Nodes ........... 42
Partitions Can Be Replicated for Resilience ................................................... 45
A Minimum of Two Phases Are Needed to Maintain Consistency
across Partitions ................................................................................................ 46
In Distributed Systems, Ordering Cannot Depend on System
Timestamps ........................................................................................................ 49
A Consistent Core Can Manage the Membership of a Data Cluster ......... 58
Gossip Dissemination for Decentralized Cluster Management ................... 62

Part II: Patterns of Data Replication ................................................ 69

Chapter 3: Write-Ahead Log ............................................................................. 71
Problem .............................................................................................................. 71
Solution .............................................................................................................. 71
Implementation Considerations .............................................................. 73
Usage in Transactional Storage ................................................................ 74
Compared to Event Sourcing .................................................................... 76
Examples ............................................................................................................ 76

Chapter 4: Segmented Log ................................................................................ 77

Problem .............................................................................................................. 77
Solution .............................................................................................................. 77
Examples ............................................................................................................ 79

Chapter 5: Low-Water Mark .............................................................................. 81

Problem .............................................................................................................. 81
Solution .............................................................................................................. 81
Snapshot-Based Low-Water Mark ............................................................ 82
Time-Based Low-Water Mark ................................................................... 83
Examples ............................................................................................................ 83

Chapter 6: Leader and Followers ...................................................................... 85

Problem .............................................................................................................. 85
Solution .............................................................................................................. 85
Leader Election ........................................................................................... 86
Why Quorum Read/Writes Are Not Enough for Strong
Consistency Guarantees ............................................................................ 91
Examples ............................................................................................................ 92
 Contents xi

Chapter 7: HeartBeat ......................................................................................... 93

Problem .............................................................................................................. 93
Solution .............................................................................................................. 93
Small Clusters: Consensus-Based Systems .............................................. 95
Technical Considerations ......................................................................... 96
Large Clusters: Gossip-Based Protocols .................................................. 97
Examples ............................................................................................................ 98

Chapter 8: Majority Quorum ............................................................................ 99

Problem .............................................................................................................. 99
Solution ............................................................................................................ 100
Deciding on Number of Servers in a Cluster ...................................... 100
Flexible Quorums ..................................................................................... 101
Examples .......................................................................................................... 102

Chapter 9: Generation Clock .......................................................................... 103

Problem ............................................................................................................ 103
Solution ............................................................................................................ 104
Examples .......................................................................................................... 107

Chapter 10: High-Water Mark ........................................................................ 109

Problem ............................................................................................................ 109
Solution ............................................................................................................ 109
Log Truncation ........................................................................................ 112
Examples .......................................................................................................... 115

Chapter 11: Paxos ............................................................................................ 117

Problem ............................................................................................................ 117
Solution ............................................................................................................ 117
Flow of the Protocol ................................................................................ 118
An Example Key-Value Store .................................................................. 127
Flexible Paxos ........................................................................................... 132
Examples .......................................................................................................... 132

Chapter 12: Replicated Log ............................................................................. 133

Problem ............................................................................................................ 133
Solution ............................................................................................................ 133
Multi-Paxos and Raft ............................................................................... 134
Replicating Client Requests .................................................................... 135
Leader Election ......................................................................................... 141
xii Contents

Technical Considerations ........................................................................ 150

Push vs. Pull ............................................................................................. 151
What Goes in the Log? ............................................................................ 151
Examples .......................................................................................................... 158

Chapter 13: Singular Update Queue .............................................................. 159

Problem ............................................................................................................ 159
Solution ............................................................................................................ 159
Choice of the Queue .............................................................................. 164
Using Channels and Lightweight Threads ............................................ 164
Backpressure ............................................................................................ 165
Other Considerations ............................................................................. 166
Examples .......................................................................................................... 166

Chapter 14: Request Waiting List .................................................................. 167

Problem ............................................................................................................ 167
Solution ............................................................................................................ 167
Expiring Long Pending Requests ........................................................... 172
Examples .......................................................................................................... 173

Chapter 15: Idempotent Receiver ................................................................... 175

Problem ............................................................................................................ 175
Solution ............................................................................................................ 175
Expiring the Saved Client Requests ....................................................... 179
Removing the Registered Clients ........................................................... 180
At-Most-Once, At-Least-Once, and Exactly-Once Actions .................. 181
Examples .......................................................................................................... 181

Chapter 16: Follower Reads ............................................................................ 183

Problem ............................................................................................................ 183
Solution ............................................................................................................ 183
Finding the Nearest Replica .................................................................... 184
Disconnected or Slow Followers ............................................................ 187
Read Your Own Writes .......................................................................... 188
Linearizable Reads ................................................................................... 191
Examples .......................................................................................................... 191

Chapter 17: Versioned Value .......................................................................... 193

Problem ............................................................................................................ 193
 Contents xiii

Solution ............................................................................................................ 193

Ordering of Versioned Keys ................................................................... 194
Reading Multiple Versions ...................................................................... 197
MVCC and Transaction Isolation ........................................................... 199
Using RocksDB-Like Storage Engines .................................................... 200
Examples .......................................................................................................... 201

Chapter 18: Version Vector ............................................................................. 203

Problem ............................................................................................................ 203
Solution ............................................................................................................ 203
Comparing Version Vectors .................................................................... 205
Using Version Vector in a Key-Value Store .......................................... 207
Examples .......................................................................................................... 216

Part III: Patterns of Data Partitioning ........................................... 217

Chapter 19: Fixed Partitions ........................................................................... 219
Problem ............................................................................................................ 219
Solution ............................................................................................................ 220
Choosing the Hash Function .................................................................. 221
Mapping Partitions to Cluster Nodes .................................................... 222
Alternative Solution: Partitions Proportional to Number of
Nodes ......................................................................................................... 236
Examples .......................................................................................................... 241

Chapter 20: Key-Range Partitions .................................................................. 243

Problem ............................................................................................................ 243
Solution ............................................................................................................ 244
Predefining Key Ranges .......................................................................... 244
An Example Scenario ............................................................................... 247
Auto-Splitting Ranges .............................................................................. 249
Examples .......................................................................................................... 255

Chapter 21: Two-Phase Commit ..................................................................... 257

Problem ............................................................................................................ 257
Solution ............................................................................................................ 257
Locks and Transaction Isolation ............................................................. 261
Commit and Rollback .............................................................................. 268
An Example Scenario ............................................................................... 273
xiv Contents

Using Versioned Value ............................................................................ 279

Using Replicated Log ............................................................................... 291
Failure Handling ....................................................................................... 291
Transactions across Heterogeneous Systems ........................................ 297
Examples .......................................................................................................... 297

Part IV: Patterns of Distributed Time ............................................. 299

Chapter 22: Lamport Clock ............................................................................. 301
Problem ............................................................................................................ 301
Solution ............................................................................................................ 301
Causality, Time, and Happens-Before ................................................... 302
An Example Key-Value Store .................................................................. 303
Partial Order ............................................................................................. 305
A Single Leader Server Updating Values .............................................. 306
Examples .......................................................................................................... 307

Chapter 23: Hybrid Clock ............................................................................... 309

Problem ............................................................................................................ 309
Solution ............................................................................................................ 309
Multiversion Storage with Hybrid Clock ............................................... 312
Using Timestamp to Read Values .......................................................... 314
Assigning Timestamp to Distributed Transactions .............................. 314
Examples .......................................................................................................... 316

Chapter 24: Clock-Bound Wait ....................................................................... 317

Problem ............................................................................................................ 317
Solution ............................................................................................................ 318
Read Restart .............................................................................................. 322
Using Clock-Bound APIs ......................................................................... 325
Examples .......................................................................................................... 332

Part V: Patterns of Cluster Management ........................................ 335

Chapter 25: Consistent Core ........................................................................... 337
Problem ............................................................................................................ 337
Solution ............................................................................................................ 337
Metadata Storage .................................................................................... 339
Handling Client Interactions ................................................................. 339
Examples .......................................................................................................... 342
 Contents xv

Chapter 26: Lease ............................................................................................. 345

Problem ............................................................................................................ 345
Solution ............................................................................................................ 345
Attaching the Lease to Keys in the Key-Value Storage ....................... 351
Handling Leader Failure .......................................................................... 353
Examples .......................................................................................................... 354

Chapter 27: State Watch .................................................................................. 355

Problem ............................................................................................................ 355
Solution ............................................................................................................ 355
Client-Side Implementation .................................................................... 356
Server-Side Implementation .................................................................... 356
Handling Connection Failures ................................................................ 359
Examples .......................................................................................................... 362

Chapter 28: Gossip Dissemination ................................................................. 363

Problem ............................................................................................................ 363
Solution ............................................................................................................ 363
Avoiding Unnecessary State Exchange .................................................. 368
Criteria for Node Selection to Gossip .................................................... 371
Group Membership and Failure Detection ........................................... 372
Handling Node Restarts .......................................................................... 372
Examples .......................................................................................................... 373

Chapter 29: Emergent Leader ......................................................................... 375

Problem ............................................................................................................ 375
Solution ............................................................................................................ 375
Sending Membership Updates to All the Existing Members .............. 379
An Example Scenario ............................................................................... 382
Handling Missing Membership Updates ............................................... 384
Failure Detection ...................................................................................... 385
Comparison with Leader and Followers ............................................... 392
Examples .......................................................................................................... 392

Part VI: Patterns of Communication between Nodes ..................... 393

Chapter 30: Single-Socket Channel ................................................................ 395
Problem ............................................................................................................ 395
Solution ............................................................................................................ 395
Examples .......................................................................................................... 397
xvi Contents

Chapter 31: Request Batch .............................................................................. 399

Problem ............................................................................................................ 399
Solution ............................................................................................................ 399
Technical Considerations ........................................................................ 404
Examples .......................................................................................................... 404

Chapter 32: Request Pipeline ......................................................................... 405

Problem ............................................................................................................ 405
Solution ............................................................................................................ 405
Examples .......................................................................................................... 408

References ........................................................................................................ 409

Index ................................................................................................................ 413
Foreword

Engineers are often attracted to distributed computing, which promises not only
benefits like scalability and fault tolerance but also the prestige of creating clever,
talk-worthy computer systems. But the reality is that distributed systems are hard.
There are myriads of edge cases, all with subtle interactions and high-dimensional
nuance. Every move you make as a systems designer has n-th degree side effects
which aren’t obvious. You’re Sideshow Bob, surrounded by lawn rakes, and every
step you take results in a rake in the face—until you’ve left the field or expended
all the rakes. (Oh, and even when you’ve left the field, there’s still a rake or two
waiting to be trodden on.)
So how do we avoid, or at least minimize, these pitfalls? The traditional ap-
proach has been to accept that distributed systems theory and practice are both
hard, and to work your way through textbooks and academic papers with confus-
ing or playful titles, studying numerous proofs so that you can carve out small
areas of relative safety and expertise within which to build your system. There’s
a lot of value in that approach for those that can stay the course. Systems profes-
sionals who have grown up that way seem to have a knack for spotting trouble
far down the line, and possess a good deal of technical background for reasoning
about how to solve problems—or at least minimize their likelihood or impact.
However, in other areas of software engineering, this kind of educational hazing
is not so commonplace. Instead of being thrown in at the deep end, we use ab-
stractions to help us gradually learn at greater levels of detail, from higher to
lower levels of abstraction, which often maps neatly onto the way software is
designed and built. Abstractions allow us to reason about behaviors without
getting bogged down in implementation complexity. In a distributed system where
complexity is high, some abstractions can be very useful.
In general software engineering, design patterns are a common abstraction. A
design pattern is a standardized solution to a recurrent problem in software design.
Patterns provide a language that practitioners use to reason about and discuss
problems in a well-understood manner. For example, when someone asks, “How
does this work?” you may hear something like, “It’s just a visitor.” Such exchanges,

xvii
xviii Foreword

based on a shared understanding of named patterns that solve common problems,

are short and information-rich.
The notion of taking something complex and abstracting it into a pattern is
both important and fundamental to this book. It applies the pattern approach to
the essential building blocks of modern distributed systems, naming the compo-
nents and describing their behaviors and how they interact. In doing so, it equips
you with a pattern language that, within reason, lets you treat a distributed system
as a set of composable Lego blocks.
Now, you can talk about “a system that depends on a replicated log with quo-
rum commits” without getting bogged down in the specific details of the data
structures and consensus algorithms. Perhaps more importantly, it minimizes the
risk of talking past one another because in distributed systems, textbook
terms—such as “consistency”—often have several meanings depending on context.
The effect is liberating for practitioners who now have an expressive common
vocabulary to expedite and standardize communication. But it’s also liberating
for learners who can take a structured, breadth-first tour of distributed systems
fundamentals, tackling a pattern at a time and observing how those patterns in-
teract or depend on one another. You can also, where needed, go deep into the
implementation—this book does not shy away from implementation details either.
My hope is that the patterns in this book will help you teach, learn, and com-
municate more effectively about distributed systems. It will certainly help you
avoid some of the lawn rakes.

—Jim Webber, Chief Scientist, Neo4j

Preface

Why This Book

In 2017, I was involved in developing a software system for a large optical tele-
scope called Thirty Meter Telescope (TMT). We needed to build a core framework
and services to be used by various subsystems. The subsystem components had
to discover each other and detect component failures. There was also a require-
ment to store metadata about these components. The service responsible for
storing this information had to be fault-tolerant. We couldn’t use off-the-shelf
products and frameworks due to the unique nature of the telescope ecosystem.
We had to build it all from scratch—to create a core framework and services that
different subsystems of the software could use. In essence, we had to build a
distributed system.
I had designed and architected enterprise systems using products such as Kafka,
Cassandra, and MongoDB or cloud services from providers like AWS and GCP.
All these products and services are distributed and solve similar problems. For
the TMT system, we had to build a solution ourselves. To compare and validate
our implementation choices with these proven products, we needed a deeper
understanding of the inner workings of some of these products. We had to figure
out how all these cloud services and products are built and why they are built
that way. Their own documentation often proved too product-specific for that.
Information about how distributed systems are built is scattered across various
research papers and doctoral theses. However, these academic sources have their
limitations too. They tend to focus on specific aspects, often making only passing
references to related topics. For instance, consider a well-written thesis, Consensus:
Bridging Theory and Practice [Ongaro2014]. It thoroughly explains how to implement
the Raft consensus algorithm. But you won’t know how Raft is used by products
like etcd for tracking group membership and related metadata for other products,
such as Kubernetes. Leslie Lamport’s famous paper “Time, Clocks, and the Order-
ing of Events in a Distributed System” [Lamport1978] talks about how to use

xix
xx Preface

logical clocks—but you won’t know how products like MongoDB use them as a
version for the data they store.
I believe that writing code is the best way to test your understanding. Martin
Fowler often says, “Code is like the mathematics of our profession. It’s where we
have to remove the ambiguity.” So, to get a deeper understanding of the building
blocks of distributed systems, I decided to build miniature versions of these
products myself. I started by building a toy version of Kafka. Once I had a rea-
sonable version, I used it to discuss some of the concepts of distributed systems.
That worked well. To verify that explaining concepts through code works effec-
tively, I conducted a few workshops within my company, Thoughtworks. Those
turned out to be very useful. So I extended this to products like Cassandra,
Kubernetes, Akka, Hazelcast, MongoDB, YugabyteDB, CockroachDB, TiKV, and
Docker Swarm. I extracted code snippets to understand the building blocks of
these products. Not surprisingly, there were a lot of similarities in these building
blocks. I happened to discuss this with Martin Fowler a few years back, and he
suggested writing about these as patterns. This book is the outcome of my work
with Martin to document these common building blocks of distributed system
implementations as patterns.

Who This Book Is For

Software architects and developers today face a plethora of choices when it comes
to selecting products and cloud services that are distributed by design. These
products and services claim to make certain implementation choices. Understand-
ing these choices intuitively can be challenging. Just reading through the docu-
mentation is not enough. Consider sentences like “AWS MemoryDB ensures
durability with a replicated transactional log” or “Apache Kafka operates indepen-
dently from ZooKeeper” or “Google Spanner provides external consistency with
accurate timing maintained by TrueTime.” How do you interpret these?
To get better insights, professionals rely on certifications from product providers.
But most certifications are very product-specific. They focus only on the surface
features of the product but not the underlying technical principles. Professional
developers need to have an intuitive understanding of technical details that are
specific enough to be expressed at the source-code level but generic enough to
apply to a wide range of situations. Patterns help there. Patterns in this book will
enable working professionals to have a good idea of what’s happening under the
hood of various products and services and thus make informed and effective
choices.
I expect most readers of this book to be in this group. In addition to those who
work with existing distributed systems, however, there is another group of readers
 Preface xxi

who must build their own distributed systems. I hope the patterns in this book
will give that other group a head start. There are numerous references to design
alternatives used by various products, which might be useful to these readers.

A Note on Examples
I have provided code examples for most of the patterns. The code examples are
based on my own miniature implementations of the various products I studied
while working through these patterns. My choice of language is based on what
I think most readers are likely to be able to read and understand. Java is a good
choice here. The code examples use a minimum of Java language features—mostly
methods and classes, which are available in most programming languages.
Readers familiar with other programming languages should be able to easily un-
derstand these code examples. This book, however, is not intended to be specific
for any particular software platform. Once you understand the code examples,
you will find similarities in code bases in C++, Rust, Go, Scala, or Zig. My hope
is that, once you are familiar with the code examples and the patterns, you will
find it easier to navigate the source code of various open-source products.

How to Read This Book

The book has six numbered parts that are divided into two main conceptual
sections.
First, a number of narrative chapters cover the essential topics in distributed
systems design. These chapters (in Part I) present challenges in distributed system
design along with their solutions. However, they don’t go into much detail on
these solutions.
Detailed solutions, structured as patterns, are provided in the second section
of the book (Parts II to VI). The patterns fall into four main categories: replication,
partitioning, cluster management, and network communication. Each of these is
a key building block of a distributed system.
Consider these patterns as references; there’s no need to read them cover to
cover. You may read the narrative chapters for an overview of the book’s scope,
and then explore the patterns based on your interests and requirements.
For additional reference materials, visit https://martinfowler.com/articles/patterns-of-
distributed-systems.
I hope these patterns will assist fellow software professionals in making
informed decisions in their daily work.
xxii Preface

Register your copy of Patterns of Distributed Systems on the InformIT site for
convenient access to updates and/or corrections as they become available. To
start the registration process, go to informit.com/register and log in or create an
account. Enter the product ISBN (9780138221980) and click Submit. Look on the
Registered Products tab for an Access Bonus Content link next to this product,
and follow that link to access any available bonus materials. If you would like
to be notified of exclusive offers on new editions and updates, please check the
box to receive email from us.
Acknowledgments

First and foremost, the book was only possible because of encouragement from
Martin Fowler. He guided me to think in terms of patterns. He also helped me
come up with good examples and contributed to the chapters that were very
tricky to write.
I want to thank the Thirty Meter Telescope (TMT) team. Working with that
team was the trigger for much of this work. I had good conversations about many
of these patterns with Mushtaq Ahmed who was leading the TMT project.
Sarthak Makhija validated a lot of these patterns while he worked on building
a distributed key-value store.
I have been publishing these patterns periodically on martinfowler.com. While
working on these patterns, I sent drafts of new material to the Thoughtworks
developer mailing list and asked for feedback. I want to thank the following
people for posting their feedback on the mailing list: Rebecca Parsons, Dave
Elliman, Samir Seth, Prasanna Pendse, Santosh Mahale, James Lewis, Chris Ford,
Kumar Sankara Iyer, Evan Bottcher, Ian Cartwright, and Priyanka Kotwal. Jojo
Swords, Gareth Morgan, and Richard Gall from Thoughtworks helped with
copyediting the earlier versions published on martinfowler.com.
While working on the patterns, I interacted with many people. Professor Indranil
Gupta provided feedback on the Gossip Dissemination pattern. Dahlia Malkhi
helped with questions about Google Spanner. Mikhail Bautin, Karthik Ran-
ganathan, and Piyush Jain from the Yugabyte team answered all my questions
about some of implementation details in YugabyteDB. The CockroachDB team
was very responsive in answering questions about their design choices. Bela Ban,
Patrik Nordwall, and Lalith Suresh provided good feedback on the Emergent
Leader pattern.
Salim Virji and Jim Webber went through the early manuscript and provided
some nice feedback. Richard Sites provided some nice suggestions on the first
chapter. I want to extend my heartfelt thanks to Jim Webber for contributing the
foreword to this book.
One of the great things about being an employee at Thoughtworks is that they
allowed me to spend considerable time on this book. Thanks to the Engineering

xxiii
xxiv Acknowledgments

for Research (E4R) group of Thoughtworks for their support. I want to also thank
Sameer Soman, MD, Thoughtworks India, who always encouraged me.
At Pearson, Greg Doench is my acquisition editor, navigating many issues in
getting a book to publication. I was glad to work with Julie Nahil as my production
editor. It was great to work with Dmitry Kirsanov for copyediting and Alina
Kirsanova for composition and indexing.
My family has been a source of constant support. My mother was always very
hopeful about the book. My wife, Ashwini, is an excellent software developer
herself; she and I had insightful discussions and she provided valuable reviews
of early drafts. My daughter, Rujuta, and son, Advait, were sources of my
motivation.
About the Author

Unmesh Joshi is a Principal Consultant at Thoughtworks with 24 years of industry

experience. As an ardent enthusiast of software architecture, he firmly believes
that today’s tech landscape requires a profound understanding of distributed
systems principles. For the last three years he has been publishing patterns of
distributed systems on martinfowler.com. He has also conducted various training
sessions around this topic. You can find him on X (formerly Twitter): @unmeshjoshi.

xxv
This page intentionally left blank
Part I

Narratives

1
This page intentionally left blank
Chapter 1

The Promise and Perils of

Distributed Systems

The Limits of a Single Server

In this book, we will discuss distributed systems. But what exactly do we mean
when we say “distributed systems”? And why is distribution necessary? Let’s start
from the basics.
In today’s digital world, the majority of our activities rely on networked services.
Whether it’s ordering food or managing finances, these services run on servers
located somewhere. When using cloud services like AWS, GCP, or Azure, these
servers are managed by the respective cloud providers. They store data, process
user requests, and perform computations using the CPU, memory, network, and
disks. These four fundamental physical resources are essential for any computation.
Consider a typical retail application functioning as a networked service, where
users can perform actions such as adding items to their shopping cart, making
purchases, viewing orders, and querying past orders. The capacity of a single
server to handle user requests is ultimately determined by the limitations of four
key resources: network bandwidth, disks, CPU, and memory.
The network bandwidth sets the maximum data transfer capacity over the
network at any given time. For example, with a network bandwidth of 1Gbps
(125MB/s) and 1KB records being written or read, the network can support a
maximum of 125,000 requests per second. However, if the record size increases
to 5KB, the number of requests that can be passed over the network decreases to
only 25,000.
Disk performance depends on several factors, including the type of read or
write operations and how well disk caches are used. Mechanical disks are also
affected by hardware features such as rotational speed and seek time. Sequential
operations usually have better performance than random ones. Moreover, the
performance is influenced by concurrent read/write operations and software-
based transactional processes. These factors can significantly affect the overall
throughput and latency on a single server.

3
4 Chapter 1 The Promise and Perils of Distributed Systems

Figure 1.1 Resources of computation

Likewise, when the CPU or memory limit is reached, requests must wait for
their turn to be processed. When these physical limits are pushed to their capac-
ity, this results in queuing. As more requests pile up, waiting times increase,
negatively impacting the server’s ability to efficiently handle user requests.

Figure 1.2 Drop in throughput with increase in requests

 Separate Business Logic and Data Layer 5

The impact of reaching the limits of these resources becomes evident in the
overall throughput of the system, as illustrated in Figure 1.2.
This poses a problem for end users. As the system is expected to accommodate
an increasing user base, its performance actually degrades.
To ensure requests are served effectively, you have to divide and process them
on multiple servers. This enables the utilization of separate CPUs, networks,
memory, and disks to handle user requests. In our example, the workload should
be divided so that each server handles approximately five hundred requests.

Separate Business Logic and Data Layer

A common approach is to separate an architecture into two parts. The first part
is the stateless component responsible for exposing functionality to end users.
This can take the form of a web application or, more commonly, a web API that
serves user-facing applications. The second part is the stateful component, which
is managed by a database (Figure 1.3).

Figure 1.3 Separating compute and data

This way, most of the application logic executes on the separate server utilizing
a separate network, CPU, memory, and disk. This architecture works particularly
well if most users can be served from caches put at different layers in the archi-
tecture. It makes sure that only a portion of all requests need to reach the database
layer.
As the number of user requests increases, more servers can be added to handle
the stateless business logic. This scalability allows the system to accommodate a
growing user base and ensures that requests can be processed efficiently. In the
event of a server failure, a new server can be introduced to take over the workload
and continue serving user requests seamlessly (Figure 1.4).
6 Chapter 1 The Promise and Perils of Distributed Systems

Figure 1.4 Scaling compute with multiple servers

This approach is effective for many applications. However, there comes a point
when the amount of data stored in stateful databases grows to hundreds of tera-
bytes or even petabytes, or the number of requests to the database layer increases
significantly. As a result, the simplistic architecture described above runs into
limitations stemming from the physical constraints of the four fundamental
resources on the server responsible for managing the data.

Partitioning Data
When a software system runs into hardware’s physical limits, the best approach
to ensure proper request processing is to divide the data and process it on mul-
tiple servers (Figure 1.5). This enables the utilization of separate CPUs, networks,
memory, and disks to handle requests on smaller data portions.
 A Look at Failures 7

Figure 1.5 Scaling data by distributing on multiple servers

A Look at Failures
When we utilize multiple machines with their own disk drives, network intercon-
nects, processors, and memory units, the likelihood of failures becomes a signifi-
cant concern. Consider the hard disk failure probability. If a disk has a failure
rate of once in 1000 days, the probability of it failing on any given day is 1/1000,
which may not be a major concern on its own. However, if we have 1000 disks,
the probability of at least one disk failing on a given day becomes 1. If the parti-
tioned data is being served from the disk that fails, it will become unavailable
until the disk is recovered.
To gain insights into the types of failures that can occur look at the failure
statistics from Jeff Dean’s 2009 talk [Dean2009] on Google’s data centers as shown
in Table 1.1. Although these numbers are from 2009, they still provide a valuable
representation of failure patterns.
8 Chapter 1 The Promise and Perils of Distributed Systems

Table 1.1 Failure Events per Year for a Cluster in a Data Center from Jeff Dean’s 2009 Talk
[Dean2009]

Event Details

Overheating Power down most machines in < 5 min (~1–2 days to

recover)

PDU Failure ~500–1000 machines suddenly disappear (~6 hours to

come back)

Rack Move Plenty of warning, ~500–1000 machines powered down

(~6 hours)

Network Rewiring Rolling ~5% of machines down over 2-day span

Rack Failures 40–80 machines instantly disappear (1–6 hours to

get back)

Racks Go Wonky 40–80 machines see 50% packet loss

Network Maintenances 4 might cause ~30-minute random connectivity losses

Router Reloads Takes out DNS and external VIPs for a couple minutes

Router Failures Have to immediately pull traffic for an hour

Minor DNS Blips Dozens of 30-second blips for DNS

Individual Machine Failures 1000 individual machine failures

Hard Drive Failures Thousands of hard drive failures

When distributing stateless compute across multiple servers, failures can be

managed relatively easily. If a server responsible for handling user requests fails,
the requests can be redirected to another server, or a new server can be added
to take over the workload. Since stateless compute does not rely on specific data
stored on a server, any server can begin serving requests from any user without
the need to load specific data beforehand.
Failures become particularly challenging when dealing with data. Creating a
separate instance on a random server is not as straightforward. It requires careful
consideration to ensure that the servers start in the correct state and coordinate
with other nodes to avoid serving incorrect or stale data. This book mainly focuses
on systems that face these types of challenges.
To ensure that the system remains functional even if certain components are
experiencing failures, simply distributing data across cluster nodes is often
insufficient. It is crucial to effectively mask the failures.
 Replication: Masking Failures 9

Replication: Masking Failures

Replication plays a crucial role in masking failures and ensuring service availabil-
ity. If data is replicated on multiple machines, even in the event of failures, clients
can connect to a server that holds a copy of the data.
However, doing this is not as simple as it sounds. The responsibility for
masking failures falls on the software that handles user requests. The software
must be able to detect failures and ensure that any inconsistencies are not visible
to the users. Understanding the types of errors that a software system experiences
is vital for successfully masking these failures.
Let’s look at some of the common problems that software systems experience
and need to mask from the users of the system.

Process Crash
Software processes can crash unexpectedly due to various reasons. It could be a
result of hardware failures or unhandled exceptions in the code. In containerized
or cloud environments, monitoring software can automatically restart a process
it recognizes as faulty. However, if a user has stored data on the server and re-
ceived a successful response, it becomes crucial for the software to ensure that
the data remains available after the process restarts. Measures need to be in place
to handle process crashes and ensure data integrity and availability.

Network Delay
The TCP/IP network protocol operates asynchronously, meaning it does not
provide a guaranteed upper bound on message delivery delay. This poses a
challenge for software processes that communicate over TCP/IP. They must de-
termine how long to wait for responses from other processes. If a response is
not received within the designated time, they need to decide whether to retry or
consider the other process as failed. This decision-making becomes crucial for
maintaining the reliability and efficiency of communication between processes.

Process Pause
During the execution of a process, it can pause at any given moment. In garbage-
collected languages like Java, execution can be interrupted by garbage collection
pauses. In extreme cases, these pauses can last tens of seconds. As a result, other
processes need to determine whether the paused process has failed. The situation
becomes more complex when the paused process resumes and begins sending
messages to other processes. The other processes then face a dilemma: Should
they ignore the messages or process them, especially if they had previously
10 Chapter 1 The Promise and Perils of Distributed Systems

marked the paused process as failed? Finding the right course of action in these
circumstances is a challenging problem.

Unsynchronized Clocks
The clocks in the servers typically utilize quartz crystals. However, the oscillation
frequency of a quartz crystal can be influenced by factors like temperature changes
or vibrations. This can cause the clocks on different servers to have different
times. Servers typically require a service such as NTP1 that continuously synchro-
nizes their clocks with time sources over the network. However, network faults
can disrupt this service, leading to unsynchronized clocks on servers.2 As a result,
when processes need to order messages or determine the sequence of saved data,
they cannot rely on the system timestamps because clock timings across servers
can be inconsistent.

Defining the Term “Distributed Systems”

We will explore the common solutions to address the challenges posed by these
failures. However, before we delve into that, let’s establish a definition for
distributed systems based on our observations thus far.
A distributed system is a software architecture that consists of multiple inter-
connected nodes or servers working together to achieve a common goal. These
nodes communicate with each other over a network and coordinate their actions
to provide a unified and scalable computing environment.
In a distributed system, the workload is distributed across multiple servers, al-
lowing for parallel processing and improved performance. The system is designed
to handle large amounts of data and accommodate a high number of concurrent
users. Most importantly, it offers fault tolerance and resilience by replicating data
and services across multiple nodes, ensuring that the system remains operational
even in the presence of failures or network disruptions.

The Patterns Approach

Professionals seeking practical advice need an intuitive understanding of these
systems that goes beyond theory. They need detailed and specific explanations

1. Network Time Protocol.

2. Even Google’s TrueTime clock machinery built using GPS clocks has clock skew.
However, that clock skew has a guaranteed upper bound.
 The Patterns Approach 11

that help comprehend real code while remaining applicable to a wide range of
systems. The Patterns approach is an excellent tool to fulfill these requirements.
The concept of patterns was initially introduced by architect Christopher
Alexander in his book A Pattern Language [Alexander1977]. This approach gained
popularity in the software industry, thanks to the influential book widely known
as the Gang Of Four [Gamma1994] book.
Patterns, as a methodology, describe particular problems encountered in soft-
ware systems, along with concrete solution structures that can be demonstrated
by real code. One of the key strengths of patterns lies in their descriptive names
and the specific code-level details they provide.
A pattern, by definition, is a “recurring solution” to a problem within a specific
context. Therefore, something is only referred to as a pattern if it is
observed repeatedly in multiple implementations. Generally, The Rule of Three3 is
followed—a pattern should be observed in at least three systems before it can be
recognized as a pattern.
The patterns approach, employed in this book, is rooted in the study of actual
codebases from various open source projects, such as Apache Kafka,4
Apache Cassandra,5 MongoDB,6 Apache Pulsar,7 etcd,8 Apache ZooKeeper,9
CockroachDB,10 YugabyteDB,11 Akka,12 JGroups,13 and others. These patterns
are grounded in practical examples and can be applied to different software sys-
tems. By exploring the insights gained from these codebases, readers can learn
to understand and apply these patterns to solve common software challenges.
Another important aspect of patterns is that they are not used in isolation but
rather in conjunction with other patterns. Understanding how the patterns interlink
makes it much easier to grasp the overall architecture of the system.
The next chapter takes a tour of most of the patterns and shows how they link
together.

3. https://wiki.c2.com/?RuleOfThree
4. https://kafka.apache.org
5. https://cassandra.apache.org
6. https://www.mongodb.com
7. https://pulsar.apache.org
8. https://etcd.io
9. https://zookeeper.apache.org
10. https://www.cockroachlabs.com
11. https://www.yugabyte.com
12. https://akka.io
13. http://www.jgroups.org
This page intentionally left blank
Chapter 2

Overview of the Patterns

by Unmesh Joshi and Martin Fowler

As discussed in the last chapter, distributing data means at least one of two
things: partitioning and replication. To start our journey through the patterns in
this book, we’ll focus on replication first.
Imagine a very minimal data record that captures how many widgets we have
in four locations (Figure 2.1).

Figure 2.1 An example data record

We replicate it on three nodes: Jupiter, Saturn, and Neptune (Figure 2.2).

Figure 2.2 Replicated data record

13
14 Chapter 2 Overview of the Patterns

Keeping Data Resilient on a Single Server

The first area of potential inconsistency appears with no distribution at all. Con-
sider a case where the data for Boston, London, and Pune are held on different
files. In this case, performing a transfer of 40 widgets means changing bos.json to
reduce its count to 10 and changing pnq.json to increase its count to 115. But what
happens if Neptune crashes after changing Boston’s file but before updating
Pune’s? In that case we would have inconsistent data, destroying 40 widgets
(Figure 2.3).

Figure 2.3 Node crash causes inconsistency

An effective solution to this is Write-Ahead Log (Figure 2.4). With this, the
message handler first writes all the information about the required update to a

Figure 2.4 Using WAL

 Competing Updates 15

log file. This is a single write, so is simple to ensure it’s done atomically. Once the
write is done, the handler can acknowledge to its caller that it has handled
the request. Then the handler, or other component, can read the log entry and
carry out the updates to the underlying files.
Should Neptune crash after updating Boston, the log should contain enough
information for Neptune, when it restarts, to figure out what happened and restore
the data to a consistent state, as shown in Figure 2.5. (In this case it would
store the previous values in the log before any updates are made to the data file.)

Figure 2.5 Recovery using WAL

The log gives us resilience because, for a known prior state, the linear sequence
of changes determines the state after the log is executed. This property is impor-
tant for resilience in a single node scenario but, as we’ll see, it’s also very valuable
for replication. If multiple nodes start at the same state, and they all play the
same log entries, we know they will end up at the same state too.
Databases use a Write-Ahead Log, as discussed in the above example, to
implement transactions.

Competing Updates
Suppose two different users, Alice and Bob, are connecting to two different
cluster nodes to execute their requests. Alice wants to move 30 widgets from
Boston to London, while Bob wants to move 40 widgets from Boston to Pune
(Figure 2.6).
16 Chapter 2 Overview of the Patterns

Figure 2.6 Competing updates

How should the cluster resolve this? We can’t have any node just decide to do
an update because we’d quickly run into inconsistency hell as we try to figure
out how to get Boston to store antimatter widgets. One of the most straightforward
approaches is Leader and Followers, where one of the nodes is marked as the
leader, and the others are considered followers. In this situation, the leader
handles all updates and broadcasts those updates to the followers. Let’s say
Neptune is the leader in this cluster. Then, Jupiter will forward Alice’s A1 request
to Neptune (Figure 2.7).

Figure 2.7 Leader handling all the updates

Neptune now gets both update requests, so it has the sole discretion as to how
to deal with them. It can process the first one it receives (Bob’s B1) and reject
A1 (Figure 2.8).
 Dealing with the Leader Failing 17

Figure 2.8 Leader rejecting requests for insufficient widgets

Dealing with the Leader Failing

That’s what happens most of the time—when all goes well. But the point of getting
a distributed system to work is what happens when things don’t go well. Here’s a
different case. Neptune receives B1 and sends out its replication messages. But
it is unable to contact Saturn. It could replicate only to Jupiter. At this point it
loses all connectivity with the other two nodes. This leaves Jupiter and Saturn
connected together, but disconnected from their leader (Figure 2.9).
So now what do these nodes do? For a start, how do they even find out what’s
broken? Neptune can’t send Jupiter and Saturn a message saying the connection
is broken . . . because the connection is broken. Nodes need a way to find out
when connections to their colleagues break. They do this with a HeartBeat—or,
more strictly, with the absence of a heartbeat.
A heartbeat is a regular message sent between nodes, just to indicate they are
alive and communicating. Heartbeat does not necessarily require a distinct mes-
sage type. When cluster nodes are already engaged in communication, such as
when replicating data, the existing messages can serve the purpose of heartbeats.
If Saturn doesn’t receive a heartbeat from Neptune for a period of time, Saturn
marks Neptune as down. Since Neptune is the leader, Saturn now calls for an
election for a new leader (Figure 2.10).
18 Chapter 2 Overview of the Patterns

Figure 2.9 Leader failure

Figure 2.10 Leader sending heartbeats

 Dealing with the Leader Failing 19

The heartbeat gives us a way to know that Neptune has disconnected, so now
we can turn to the problem of how to deal with Bob’s request. We need to ensure
that once Neptune has confirmed the update to Bob, even if Neptune crashes,
the followers can elect a new leader with B1 applied to their data. But we also
need to deal with more complication than that, as Neptune may have received
multiple messages. Consider the case where there are messages from both Alice
(A1) and Bob (B1) handled by Neptune. Neptune successfully replicates them
both with Jupiter but is unable to contact Saturn before it crashes, as shown in
Figure 2.11.

Figure 2.11 Leader failure—incomplete replication

In this case, how do Jupiter and Saturn deal with the fact that they have different
states?
The answer is essentially the same as discussed earlier for resilience on a single
node. If Neptune writes changes into a Write-Ahead Log and treats replication as
20 Chapter 2 Overview of the Patterns

copying those log entries to its followers, then its followers will be able to figure
out what the correct state is by examining the log entries (Figure 2.12).

Figure 2.12 Leader failure—incomplete replication—using log

When Jupiter and Saturn elect a new leader, they can tell that Jupiter’s log
has later index entries, and Saturn can apply those log entries to itself to gain a
consistent state with Jupiter.
This is also why Neptune can reply to Bob that the update was accepted, even
though it hadn’t heard back from Saturn. As long as a Majority Quorum—that is,
a majority—of the nodes in the cluster have successfully replicated the log mes-
sages, Neptune can be sure that the cluster will maintain consistency even if the
leader disconnects.
 Multiple Failures Need a Generation Clock 21

Multiple Failures Need a Generation Clock

We assumed here that Jupiter and Saturn can figure out whose log is most up
to date. But things can get trickier. Let’s say Neptune accepted a request from
Bob to move 40 widgets from Boston to Pune but failed before replicating it
(Figure 2.13).

Figure 2.13 Leader fails before replication.

Jupiter is elected as a new leader, and accepts a request from Alice to move
30 widgets from Boston to London. But it also crashes before replicating the
request to other nodes (Figure 2.14).
In a while, Neptune and Jupiter come back, but before they can talk, Saturn
crashes. Neptune is elected as a leader. Neptune checks with itself and Jupiter
for the log entries. It will see two separate requests at index 1, the one from Bob
which it had accepted and the one from Alice that Jupiter has accepted. Neptune
can’t tell which one it should pick (Figure 2.15).
To solve this kind of situation, we use a Generation Clock. This is a number that
increments with each leadership election. It is a key requirement of Leader and
Followers.
Looking at the previous scenario again, Neptune was leader for generation 1.
It adds Bob’s entry in its log marking it with its generation (Figure 2.16).
When Jupiter gets elected as a leader, it increments the generation to 2. So
when it adds Alice’s entry to its log, it’s marked for generation 2 (Figure 2.17).
22 Chapter 2 Overview of the Patterns

Figure 2.14 New leader fails before replication.

Figure 2.15 Leader needs to resolve existing log entries.

 Multiple Failures Need a Generation Clock 23

Figure 2.16 Leader adds generation to log entries.

Figure 2.17 New leader increments generation.

Random documents with unrelated
content Scribd suggests to you:
evil, sadness, joy, are lost upon Steinlen. He seems to be en rapport with
those people whom we call our brothers and treat as our inferiors and our
enemies. The tricks, the attitudes, the expressions, the behaviour of the
passers-by are familiar to him. With a few clever strokes of brush or pencil
he has given us the piquante ouvrière, the modest apprentice, in a graceful,
unconscious pose as she poises on her hip the hat-box she is carrying to a
customer. In this jeune fille representative, in the toss of her head, the curve
of her arm, the swish of her neat skirts high above her well-shod feet, in this
jaunty creature, scrupulous as to tidiness, is a certain phase of Paris drawn
with crystalline distinctness. This girl of the people is dangerous, charming,
and to her the boulevardier is an enemy. Paris is epitomised by this one
flaring street-wall decoration. So much for the jeunesse, gay, laborious, and
generally self-respecting. We do not often see Steinlen so gay.
In his studio in the Rue Caulaincourt, walking to and fro, he converses
delightfully with his friends. He is a workman, his muscular hands are full
of force, his build powerful, although he is compact and small. He suggests
endurance, and the patience that is the result of hand-to-hand tussle with
existence, and tender understanding of the needs of one’s neighbour. His
eyes are keen, penetrating, and kind, his features strong. He has a golden
voice, as the French say, caressing and indolent. Its measured cadence, its
slow, agreeable flow contrasts strongly with the man himself. He wears the
coarse velveteens of the labourer, cuffs close around his muscular wrists.
His trousers are voluminous, and on his feet flap loose Indian slippers. His
appearance, and the fact that his studio is in one of the poorest quarters of
Paris, his whole attitude and life, suggest, not a
 EN ATTENDANT

scorn of material things, but a perfect ability to forego them.

The studio is a workshop first of all; it is free from would-be artistic
decorations, full of canvases and folios of drawings. He has his own
lithographic stones, and does his own printing, holding this as an art
important, as do many of the moderns; and he, as well as many another
modern, insists on the expediency of making his own colours.
Steinlen thinks unfavourably of academies and salons, “where,” he says,
“in order to exhibit at stated periods the artists paint anything and
everything under rush and stress.” He sends to no public collections. When
a goodly number of studies have accumulated in his studio, he organises a
little exposition of his own, and the admirers of his work have an
opportunity to visit a Steinlen exhibition in the autumn, when he will
delight, charm, and touch the public as he has never hitherto failed to do.
 At this period of his life—and he is still young—he finds the insistent
needs of daily existence are met; and he draws, as it were, a sigh of relief,
and turns toward what is his recreation, because a beloved labour, and the
goal of his career, painting in oil. Steinlen bemoans the fact that he lacks the
prescribed technicalities an academic education would have given to him;
but this fact has left him a freedom from rules which, in spite of their
immense importance, are often trammels to individuality. At all events the
daring boldness of his stroke, and his perfect originality, have been
developed with no hindrance. Need has been his spur, talent the response to
the goad.
Possibly Steinlen is nowhere better displayed than in a certain canvas at
present in his studio in the Rue Caulaincourt. It is a life-size oil painting, a
study of a man and woman in the working class. It is evidently the end of
the day, and the scene a nook or corner in some room so distant from the
rooms we all know that it is hard even to imagine where it may be. The
workman has taken the young creature in his arms for a long embrace. His
head is bent over her, and she looks out from the picture above the man’s
arm. Her face is exquisite, and in thorough keeping with the type of her
class.
The sombre note predominates throughout Steinlen’s work. That
inevitable penalty of sadness which must be paid when the eye dares to
look, and the soul dares to consider how our fellow beings struggle for
existence.
 A STUDY

The work of this man, who is not a caricaturist, but a student and faithful
representer, bears a strong likeness to things in literature rather than to
things in art. He suggests Dickens, Zola, Tolstoi. Throughout his work is
apparent the broad sympathy of a man of the people who has espoused their
cause and made himself their prophet.
Part of the crowd, elbow to elbow with humanity in the very vortex of
the mass, he has felt the multitude, blood and sinew, around him, until it has
become amalgamated fairly with his inspiration. Then withdrawn to a
fortification, possibly, of his city, in semi-retirement, he lets the turbulent
suggestions take form that he may present them to the world. Thus Steinlen,
so closely of the people, is in reality separated from mankind by virtue of
his talent. And if to the eye demanding agreeable form, beauty seems
sometimes lacking in this artist’s strong, original, profoundly human
creations, it may be said that æsthetics do not abound in the walks of life
which this student of humanity portrays.

Printed by Ballantyne, Hanson & Co.

London & Edinburgh

FOOTNOTE:
[A] Chez Pelletan.
 *** END OF THE PROJECT GUTENBERG EBOOK MODERN FRENCH
MASTERS ***

Updated editions will replace the previous one—the old editions will
be renamed.

Creating the works from print editions not protected by U.S.

copyright law means that no one owns a United States copyright in
these works, so the Foundation (and you!) can copy and distribute it
in the United States without permission and without paying
copyright royalties. Special rules, set forth in the General Terms of
Use part of this license, apply to copying and distributing Project
Gutenberg™ electronic works to protect the PROJECT GUTENBERG™
concept and trademark. Project Gutenberg is a registered trademark,
and may not be used if you charge for an eBook, except by following
the terms of the trademark license, including paying royalties for use
of the Project Gutenberg trademark. If you do not charge anything
for copies of this eBook, complying with the trademark license is
very easy. You may use this eBook for nearly any purpose such as
creation of derivative works, reports, performances and research.
Project Gutenberg eBooks may be modified and printed and given
away—you may do practically ANYTHING in the United States with
eBooks not protected by U.S. copyright law. Redistribution is subject
to the trademark license, especially commercial redistribution.

START: FULL LICENSE

THE FULL PROJECT GUTENBERG LICENSE
 PLEASE READ THIS BEFORE YOU DISTRIBUTE OR USE THIS WORK

To protect the Project Gutenberg™ mission of promoting the free

distribution of electronic works, by using or distributing this work (or
any other work associated in any way with the phrase “Project
Gutenberg”), you agree to comply with all the terms of the Full
Project Gutenberg™ License available with this file or online at
www.gutenberg.org/license.

Section 1. General Terms of Use and

Redistributing Project Gutenberg™
electronic works
1.A. By reading or using any part of this Project Gutenberg™
electronic work, you indicate that you have read, understand, agree
to and accept all the terms of this license and intellectual property
(trademark/copyright) agreement. If you do not agree to abide by all
the terms of this agreement, you must cease using and return or
destroy all copies of Project Gutenberg™ electronic works in your
possession. If you paid a fee for obtaining a copy of or access to a
Project Gutenberg™ electronic work and you do not agree to be
bound by the terms of this agreement, you may obtain a refund
from the person or entity to whom you paid the fee as set forth in
paragraph 1.E.8.

1.B. “Project Gutenberg” is a registered trademark. It may only be

used on or associated in any way with an electronic work by people
who agree to be bound by the terms of this agreement. There are a
few things that you can do with most Project Gutenberg™ electronic
works even without complying with the full terms of this agreement.
See paragraph 1.C below. There are a lot of things you can do with
Project Gutenberg™ electronic works if you follow the terms of this
agreement and help preserve free future access to Project
Gutenberg™ electronic works. See paragraph 1.E below.
1.C. The Project Gutenberg Literary Archive Foundation (“the
Foundation” or PGLAF), owns a compilation copyright in the
collection of Project Gutenberg™ electronic works. Nearly all the
individual works in the collection are in the public domain in the
United States. If an individual work is unprotected by copyright law
in the United States and you are located in the United States, we do
not claim a right to prevent you from copying, distributing,
performing, displaying or creating derivative works based on the
work as long as all references to Project Gutenberg are removed. Of
course, we hope that you will support the Project Gutenberg™
mission of promoting free access to electronic works by freely
sharing Project Gutenberg™ works in compliance with the terms of
this agreement for keeping the Project Gutenberg™ name associated
with the work. You can easily comply with the terms of this
agreement by keeping this work in the same format with its attached
full Project Gutenberg™ License when you share it without charge
with others.

1.D. The copyright laws of the place where you are located also
govern what you can do with this work. Copyright laws in most
countries are in a constant state of change. If you are outside the
United States, check the laws of your country in addition to the
terms of this agreement before downloading, copying, displaying,
performing, distributing or creating derivative works based on this
work or any other Project Gutenberg™ work. The Foundation makes
no representations concerning the copyright status of any work in
any country other than the United States.

1.E. Unless you have removed all references to Project Gutenberg:

1.E.1. The following sentence, with active links to, or other

immediate access to, the full Project Gutenberg™ License must
appear prominently whenever any copy of a Project Gutenberg™
work (any work on which the phrase “Project Gutenberg” appears,
or with which the phrase “Project Gutenberg” is associated) is
accessed, displayed, performed, viewed, copied or distributed:
 This eBook is for the use of anyone anywhere in the United
States and most other parts of the world at no cost and with
almost no restrictions whatsoever. You may copy it, give it away
or re-use it under the terms of the Project Gutenberg License
included with this eBook or online at www.gutenberg.org. If you
are not located in the United States, you will have to check the
laws of the country where you are located before using this
eBook.

1.E.2. If an individual Project Gutenberg™ electronic work is derived

from texts not protected by U.S. copyright law (does not contain a
notice indicating that it is posted with permission of the copyright
holder), the work can be copied and distributed to anyone in the
United States without paying any fees or charges. If you are
redistributing or providing access to a work with the phrase “Project
Gutenberg” associated with or appearing on the work, you must
comply either with the requirements of paragraphs 1.E.1 through
1.E.7 or obtain permission for the use of the work and the Project
Gutenberg™ trademark as set forth in paragraphs 1.E.8 or 1.E.9.

1.E.3. If an individual Project Gutenberg™ electronic work is posted

with the permission of the copyright holder, your use and distribution
must comply with both paragraphs 1.E.1 through 1.E.7 and any
additional terms imposed by the copyright holder. Additional terms
will be linked to the Project Gutenberg™ License for all works posted
with the permission of the copyright holder found at the beginning
of this work.

1.E.4. Do not unlink or detach or remove the full Project

Gutenberg™ License terms from this work, or any files containing a
part of this work or any other work associated with Project
Gutenberg™.

1.E.5. Do not copy, display, perform, distribute or redistribute this

electronic work, or any part of this electronic work, without
prominently displaying the sentence set forth in paragraph 1.E.1
with active links or immediate access to the full terms of the Project
Gutenberg™ License.

1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form,
including any word processing or hypertext form. However, if you
provide access to or distribute copies of a Project Gutenberg™ work
in a format other than “Plain Vanilla ASCII” or other format used in
the official version posted on the official Project Gutenberg™ website
(www.gutenberg.org), you must, at no additional cost, fee or
expense to the user, provide a copy, a means of exporting a copy, or
a means of obtaining a copy upon request, of the work in its original
“Plain Vanilla ASCII” or other form. Any alternate format must
include the full Project Gutenberg™ License as specified in
paragraph 1.E.1.

1.E.7. Do not charge a fee for access to, viewing, displaying,

performing, copying or distributing any Project Gutenberg™ works
unless you comply with paragraph 1.E.8 or 1.E.9.

1.E.8. You may charge a reasonable fee for copies of or providing

access to or distributing Project Gutenberg™ electronic works
provided that:

• You pay a royalty fee of 20% of the gross profits you derive
from the use of Project Gutenberg™ works calculated using the
method you already use to calculate your applicable taxes. The
fee is owed to the owner of the Project Gutenberg™ trademark,
but he has agreed to donate royalties under this paragraph to
the Project Gutenberg Literary Archive Foundation. Royalty
payments must be paid within 60 days following each date on
which you prepare (or are legally required to prepare) your
periodic tax returns. Royalty payments should be clearly marked
as such and sent to the Project Gutenberg Literary Archive
Foundation at the address specified in Section 4, “Information
 about donations to the Project Gutenberg Literary Archive
Foundation.”

• You provide a full refund of any money paid by a user who

notifies you in writing (or by e-mail) within 30 days of receipt
that s/he does not agree to the terms of the full Project
Gutenberg™ License. You must require such a user to return or
destroy all copies of the works possessed in a physical medium
and discontinue all use of and all access to other copies of
Project Gutenberg™ works.

• You provide, in accordance with paragraph 1.F.3, a full refund of

any money paid for a work or a replacement copy, if a defect in
the electronic work is discovered and reported to you within 90
days of receipt of the work.

• You comply with all other terms of this agreement for free
distribution of Project Gutenberg™ works.

1.E.9. If you wish to charge a fee or distribute a Project Gutenberg™

electronic work or group of works on different terms than are set
forth in this agreement, you must obtain permission in writing from
the Project Gutenberg Literary Archive Foundation, the manager of
the Project Gutenberg™ trademark. Contact the Foundation as set
forth in Section 3 below.

1.F.

1.F.1. Project Gutenberg volunteers and employees expend

considerable effort to identify, do copyright research on, transcribe
and proofread works not protected by U.S. copyright law in creating
the Project Gutenberg™ collection. Despite these efforts, Project
Gutenberg™ electronic works, and the medium on which they may
be stored, may contain “Defects,” such as, but not limited to,
incomplete, inaccurate or corrupt data, transcription errors, a
copyright or other intellectual property infringement, a defective or
damaged disk or other medium, a computer virus, or computer
codes that damage or cannot be read by your equipment.

1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except for

the “Right of Replacement or Refund” described in paragraph 1.F.3,
the Project Gutenberg Literary Archive Foundation, the owner of the
Project Gutenberg™ trademark, and any other party distributing a
Project Gutenberg™ electronic work under this agreement, disclaim
all liability to you for damages, costs and expenses, including legal
fees. YOU AGREE THAT YOU HAVE NO REMEDIES FOR
NEGLIGENCE, STRICT LIABILITY, BREACH OF WARRANTY OR
BREACH OF CONTRACT EXCEPT THOSE PROVIDED IN PARAGRAPH
1.F.3. YOU AGREE THAT THE FOUNDATION, THE TRADEMARK
OWNER, AND ANY DISTRIBUTOR UNDER THIS AGREEMENT WILL
NOT BE LIABLE TO YOU FOR ACTUAL, DIRECT, INDIRECT,
CONSEQUENTIAL, PUNITIVE OR INCIDENTAL DAMAGES EVEN IF
YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH DAMAGE.

1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you

discover a defect in this electronic work within 90 days of receiving
it, you can receive a refund of the money (if any) you paid for it by
sending a written explanation to the person you received the work
from. If you received the work on a physical medium, you must
return the medium with your written explanation. The person or
entity that provided you with the defective work may elect to provide
a replacement copy in lieu of a refund. If you received the work
electronically, the person or entity providing it to you may choose to
give you a second opportunity to receive the work electronically in
lieu of a refund. If the second copy is also defective, you may
demand a refund in writing without further opportunities to fix the
problem.

1.F.4. Except for the limited right of replacement or refund set forth
in paragraph 1.F.3, this work is provided to you ‘AS-IS’, WITH NO
OTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR ANY PURPOSE.

1.F.5. Some states do not allow disclaimers of certain implied

warranties or the exclusion or limitation of certain types of damages.
If any disclaimer or limitation set forth in this agreement violates the
law of the state applicable to this agreement, the agreement shall be
interpreted to make the maximum disclaimer or limitation permitted
by the applicable state law. The invalidity or unenforceability of any
provision of this agreement shall not void the remaining provisions.

1.F.6. INDEMNITY - You agree to indemnify and hold the Foundation,

the trademark owner, any agent or employee of the Foundation,
anyone providing copies of Project Gutenberg™ electronic works in
accordance with this agreement, and any volunteers associated with
the production, promotion and distribution of Project Gutenberg™
electronic works, harmless from all liability, costs and expenses,
including legal fees, that arise directly or indirectly from any of the
following which you do or cause to occur: (a) distribution of this or
any Project Gutenberg™ work, (b) alteration, modification, or
additions or deletions to any Project Gutenberg™ work, and (c) any
Defect you cause.

Section 2. Information about the Mission

of Project Gutenberg™
Project Gutenberg™ is synonymous with the free distribution of
electronic works in formats readable by the widest variety of
computers including obsolete, old, middle-aged and new computers.
It exists because of the efforts of hundreds of volunteers and
donations from people in all walks of life.

Volunteers and financial support to provide volunteers with the

assistance they need are critical to reaching Project Gutenberg™’s
goals and ensuring that the Project Gutenberg™ collection will
remain freely available for generations to come. In 2001, the Project
Gutenberg Literary Archive Foundation was created to provide a
secure and permanent future for Project Gutenberg™ and future
generations. To learn more about the Project Gutenberg Literary
Archive Foundation and how your efforts and donations can help,
see Sections 3 and 4 and the Foundation information page at
www.gutenberg.org.

Section 3. Information about the Project

Gutenberg Literary Archive Foundation
The Project Gutenberg Literary Archive Foundation is a non-profit
501(c)(3) educational corporation organized under the laws of the
state of Mississippi and granted tax exempt status by the Internal
Revenue Service. The Foundation’s EIN or federal tax identification
number is 64-6221541. Contributions to the Project Gutenberg
Literary Archive Foundation are tax deductible to the full extent
permitted by U.S. federal laws and your state’s laws.

The Foundation’s business office is located at 809 North 1500 West,

Salt Lake City, UT 84116, (801) 596-1887. Email contact links and up
to date contact information can be found at the Foundation’s website
and official page at www.gutenberg.org/contact

Section 4. Information about Donations to

the Project Gutenberg Literary Archive
Foundation
Project Gutenberg™ depends upon and cannot survive without
widespread public support and donations to carry out its mission of
increasing the number of public domain and licensed works that can
be freely distributed in machine-readable form accessible by the
widest array of equipment including outdated equipment. Many
small donations ($1 to $5,000) are particularly important to
maintaining tax exempt status with the IRS.

The Foundation is committed to complying with the laws regulating

charities and charitable donations in all 50 states of the United
States. Compliance requirements are not uniform and it takes a
considerable effort, much paperwork and many fees to meet and
keep up with these requirements. We do not solicit donations in
locations where we have not received written confirmation of
compliance. To SEND DONATIONS or determine the status of
compliance for any particular state visit www.gutenberg.org/donate.

While we cannot and do not solicit contributions from states where

we have not met the solicitation requirements, we know of no
prohibition against accepting unsolicited donations from donors in
such states who approach us with offers to donate.

International donations are gratefully accepted, but we cannot make

any statements concerning tax treatment of donations received from
outside the United States. U.S. laws alone swamp our small staff.

Please check the Project Gutenberg web pages for current donation
methods and addresses. Donations are accepted in a number of
other ways including checks, online payments and credit card
donations. To donate, please visit: www.gutenberg.org/donate.

Section 5. General Information About

Project Gutenberg™ electronic works
Professor Michael S. Hart was the originator of the Project
Gutenberg™ concept of a library of electronic works that could be
freely shared with anyone. For forty years, he produced and
distributed Project Gutenberg™ eBooks with only a loose network of
volunteer support.
Project Gutenberg™ eBooks are often created from several printed
editions, all of which are confirmed as not protected by copyright in
the U.S. unless a copyright notice is included. Thus, we do not
necessarily keep eBooks in compliance with any particular paper
edition.

Most people start at our website which has the main PG search
facility: www.gutenberg.org.

This website includes information about Project Gutenberg™,

including how to make donations to the Project Gutenberg Literary
Archive Foundation, how to help produce our new eBooks, and how
to subscribe to our email newsletter to hear about new eBooks.
back
back
back
back
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com