Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity

Large-Scale Architecture
The Unreasonable
Effectiveness of Simplicity
Randy Shoup
@randyshoup

• 1995 Monolithic Perl
• 1996-2002 v2
o Monolithic C++ ISAPI DLL
o 3.4M lines of code
o Compiler limits on number of methods per class
• 2002-2006 v3 Migration
o Java “mini-applications”
o Shared databases
• 2012 v4 Microservices
• 2017 v5 Microservices
@randyshoup
eBay Architecture

• 1995-2001 Obidos
o Monolithic Perl / Mason frontend over C
backend
o ~4GB application in a 4GB address space
o Regularly breaking Gnu linker
o Restarting every 100-200 requests for memory
leaks
o Releasing once per quarter
• 2001-2005 Service Migration
o Services in C++, Java, etc.
o No shared databases
• 2006 AWS launches
@randyshoup
Amazon Architecture

No one starts with microservices
…
Past a certain scale, everyone ends
up with microservices
@randyshoup

Large-Scale Architecture
•Simple Components
•Simple Interactions
•Simple Changes
•Putting It All Together

“There are two methods in
software design: One is to make
the program so simple, there are
obviously no errors. The other is
to make it so complicated, there
are no obvious errors.”
-- Tony Hoare
@randyshoup

Modular Services
• Service boundaries match the problem
domain
• Service boundaries encapsulate
business logic and data
o All interactions through published service interface
o Interface hides internal implementation details
o No back doors
• Service boundaries encapsulate
architectural -ilities
o Fault isolation
o Performance optimization
o Security boundary
@randyshoup

Orthogonal Domain Logic
• Stateless domain logic
o Ideally stateless pure function
o Matches domain problem as directly as possible
o Deterministic and testable in isolation
o Robust to change over time
• “Straight-line processing”
o Straightforward, synchronous, minimal branching
• Separate domain logic from I/O
o Hexagonal architecture, Ports and Adapters
o Functional core, imperative shell
@randyshoup

Sharding
• Shards partition the service’s “data space”
o Units for distribution, replication, processing, storage
o Hidden as internal implementation detail
• Shards encapsulate architectural -ilities
o Resource isolation
o Fault isolation
o Availability
o Performance
• Shards are autoscaled
o Divide or scale out as processing or data needs increase
o E.g., DynamoDB partitions, Aurora segments, Bigtable
tablets
@randyshoup

Service Graph
• Common services provide and
abstract widely-used capabilities
• Service topology
o Services call others, which call others, etc.
o Graph, not a strict layering
• Simplifies concerns for service team
o Concentrate on business logic
o Abstract away details of dependencies
o Focus on services you need, capabilities you
provide
@randyshoup

Common Platform
• “Paved Road”
o Shared infrastructure
o Standard frameworks
o Developer experience
o E.g., Netflix, Google
• Separation of Concerns
o Reduce cognitive load on stream-aligned
teams
o Bound decisions through enabling constraints
@randyshoup

Large-scale organizations often
invest more than 50% of
engineering effort in platform
capabilities
@randyshoup

Evolving Services
• Variation and Natural Selection
o Create / extract new services when needed to
solve a problem
o Services justify their continued existence through
usage
o Deprecate services when they are no longer used
• Services grow and evolve over time
o Factor out common libraries and services as
needed
o Teams and services split like “cellular mitosis”
@randyshoup

“Every service at
Google is either
deprecated or not
ready yet.”
@randyshoup

“Everything should
be made as simple
as possible, but not
simpler.”
@randyshoup

Event-Driven
• Communicate state changes as
stream of events
o Statement that some interesting thing
occurred
o Ideally represents a semantic domain event
• Decouples domains and teams
o Abstracted through a well-defined
interface
o Asynchronous from one another
• Simplifies component
implementation
@randyshoup

Immutable Log
• Store state as immutable log of events
o Event Sourcing
• Often matches domain
o E.g., Stitch Fix order processing / delivery state
• Log encapsulates architectural –ilities
o Durable
o Traceable and auditable
o Replayable
o Explicit and comprehensible
• Compact snapshots for efficiency
@randyshoup

Immutable Log
• Stitch Fix order states
Request a fix fix_scheduled
Assign fix to warehouse fix_hizzy_assigned
Assign fix to a stylist fix_stylist_assigned
Style the fix fix_styled
Pick the items for the fix fix_picked
Pack the items into a box fix_packed
Ship the fix via a carrier fix_shipped
Fix travels to customer fix_delivered
Customer decides, pays fix_checked_out

Embrace Asynchrony
• Decouples operations in time
o Decoupled availability
o Independent scalability
o Allows more complex processing, more
processing in parallel
o Safer to make independent changes
• Simplifies component
implementation
@randyshoup

Embrace Asynchrony
• Invert from synchronous call
graph to async dataflow
o Exploit asymmetry between writes and
reads
o Can be orders of magnitude less
resource intensive
@randyshoup

“A complex system that works is
invariably found to have evolved
from a simple system that
worked.”
-- Gall’s Law
@randyshoup

Incremental Change
• Decompose every large change into small
incremental steps
• Each step maintains backward / forward
compatibility of data and interfaces
• Multiple service versions commonly coexist
o Every change is a rolling upgrade
o Transitional states are normal, not exceptional

Continuous Testing
• Tests help us go faster
o Tests are “solid ground”
o Tests are the safety net
• Tests make better code
o Confidence to break things
o Courage to refactor mercilessly
• Tests make better systems
o Catch bugs earlier, fail faster
@randyshoup

Developer Productivity
• 75% reading
existing code
• 20% modifying
existing code
• 5% writing new
code
https://blogs.msdn.microsoft.com/peterhal/2006/01/04/what-do-programmers-really-do-anyway-aka-part-2-of-the-yardstick-saga/
@randyshoup

Continuous Testing
• Tests make better designs
o Modularity
o Separation of Concerns
o Encapsulation
@randyshoup

“There’s a deep synergy between
testability and good design. All of the
pain that we feel when writing unit tests
points at underlying design problems.”
@randyshoup
-- Michael Feathers

Test-Driven Development
• è Basically no bug tracking system (!)
o “Inbox Zero” for bugs
o Bugs are fixed as they come up
o Backlog contains features we want to build
o Backlog contains technical debt we want to repay
@randyshoup

Canary Deployments
• Staged rollout
o Go slowly at first; go faster when you gain confidence
• Automated rollout / rollback
o Automatically monitor changes to metrics
o If metrics look good, keep going; if metrics look bad, roll back
• Make deployments routine and boring
@randyshoup

Feature Flags
• Configuration “flag” to enable / disable a feature
for a particular set of users
o Independently discovered at eBay, Facebook, Google, etc.
• More solid systems
o Decouple feature delivery from code delivery
o Rapid on and off
o Separate experiment and control groups
o Develop / test / verify in production
@randyshoup

Continuous Delivery
• Deploy services multiple times per day
o Robust build, test, deploy pipeline
o SLO monitoring
o Synthetic monitoring
• More solid systems
o Release smaller, simpler units of work
o Smaller changes to roll back or roll forward
o Faster to repair, easier to understand, simpler to diagnose
o Increase rate of change and reduce risk of change
@randyshoup

• Cross-company Velocity Initiative to
improve software delivery
o Think Big, Start Small, Learn Fast
o Iteratively identify and remove bottlenecks for teams
o “What would it take to deploy your application every
day?”
• Doubled engineering productivity
o 5x faster deployment frequency
o 5x faster lead time
o 3x lower change failure rate
o 3x lower mean-time-to-restore
• Prerequisite for large-scale architecture
changes
@randyshoup
Continuous Delivery

System of Record
• Single System of Record
o Every piece of data is owned by a single service
o That service is the canonical system of record for that data
• Every other copy is a read-only, non-authoritative cache
customer-service
styling-service
customer-search
billing-service
@randyshoup

Shared Data
Option 1: Synchronous Lookup
o Customer service owns customer data
o Fulfillment service calls customer service in real time
fulfillment-service
customer-service
@randyshoup

Shared Data
Option 2: Async event + local cache
o Customer service owns customer data
o Customer service sends address-updated event when customer address
changes
o Fulfillment service caches current customer address
fulfillment-service
customer-service
@randyshoup

Joins
Option 1: Join in Client Service
o Get a single customer from customer-service
o Query matching orders for that customer from order-service
Customers
Orders
order-history-page
customer-service order-service
@randyshoup

Joins
Option 2: Service that “Materializes the View”
o Listen to events from customer-service, events from order-service
o Maintain denormalized join of customer data and orders together in local
storage
Customer Orders
customer-order-service
customer-service
order-service
@randyshoup

Netflix Viewing History
• Store and process member’s playback
data
o 1M requests per second
o Used for viewing history, personalization,
recommendations, analytics, etc.
• Original synchronous architecture
o Synchronously write to persistent storage and lookup
cache
o Availability and data loss from backpressure at high
load
• Asynchronous rearchitecture
o Write to durable queue
o Async pipeline to enrich, process, store, serve
o Materialize views to serve reads
@randyshoup Sharma Podila, 2021, Microservices to Async Processing Migration at Scale, QConPlus 2021.

Walmart Item Availability
• Is this item available to ship to this customer?
o Customer SLO 99.98% uptime in 300ms
• Complex logic involving many teams and
domains
o Inventory, reservations, backorders, eligibility, sales caps, etc.
• Original synchronous architecture
o Graph of 23 nested synchronous service calls in hot path
o Any component failure invalidates results
o Service SLOs 99.999% uptime with 50ms marginal latency
o Extremely expensive to build and operate
@randyshoup Scott Havens, 2019, Fabulous Fortunes, Fewer Failures, and Faster Fixes from Functional Fundamentals, DOES 2019.

• Invert each service to use async events
o Event-driven “dataflow”
o Idempotent processing
o Event-sourced immutable log
o Materialized view of data from upstream dependencies
• Asynchronous rearchitecture
o 2 services in synchronous hot path
o Async service SLOs 99.9% uptime with latency in seconds
or minutes
o More resilient to delays and outages
o Orders of magnitude simpler to build and operate

Thank you!
@randyshoup
linkedin.com/in/randyshoup
medium.com/@randyshoup

Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity

More Related Content

What's hot (20)

Similar to Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity (20)

More from Randy Shoup (20)

Recently uploaded (20)

Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity