Software Architecture over ZeroMQ

Software
Architecture
using ØMQ

by
Pieter Hintjens
Strange Loop 2012
Photos by Pieter Hintjens
cc-by-sa © 2012 Pieter Hintjens

A complex
story is best
told as a
series of
vacuous 1-
liners


90% of
software is
trash.

90% of the
rest will be
trash RSN


We basically
don't know
how to
make code
that can
survive ten,
let alone 50
years


The most
difficult
challenge
in our
profession
is simple
accuracy


Future code
has to talk
to code, has
to be chatty,
sociable,
well-
connected


When we
can move
faster,
where we
go is more
critical than
ever.


Writing
distributed
code is like
a live jam
session.

It's all about
other people


How we
connect to
each other
matters
more than
who we are


The physics
of software
is the
physics of
people


Ideas are
cheap.

Execution is
the hard
part


Making
perfect
software is
easy, once
you learn
the trick
(which is
kinda hard)


Simplicity
always
beats
functionality


Problems
are not all
equal, and
most are
illusions


When you
know the
real problem
you have
done half
the work


Do nothing
that is not a
minimal,
plausible
answer to a
well-defined
problem


Every
commit
should be
shippable


Design by
removing
problems,
not adding
features


Five Steps
to Satori:
Learn,
Draw,
Divide,
Conquer,
Repeat


1.
Learn the
language
before you
write a
poem


2.
If it looks
pretty, it's
more likely
to work


3.
A good
contract is
worth a
thousands
assumptions


4.
When you
take small
steps, it
hurts less
when you
fall


5.
Solve one
problem,
and repeat
until you run
out of time
or money


Distributed
software
lives or dies
by its
protocols


Protocols
are contracts
that describe
the rights
and
obligations
of each party


An unprotocol
takes minutes
to explain,
hours to
design, days to
write, weeks to
prove, months
to mature, and
years to
replace

Use human nom-protocol =
open-peering
language *use-peering
in your open-peering =
unprotocols. C:OHAI
( S:OHAI-OK / S:WTF )

use-peering =
ORLY? C:ICANHAZ
YARLY! / S:CHEEZBURGER
/ C:HUGZ S:HUGZ-OK
/ S:HUGZ C:HUGZ-OK


Use GPLv3
for your
open specs.

Remixability
is freedom


If you're
willing to
give up
flexibility for
speed you
deserve
neither
flexibility nor
speed

Use cheap
text for the
low-volume
chatty
control
commands


Use nasty
hand-coded
binary for
the high-
volume data


ØMQ
framing
makes a
lousy codec
but a great
separator


A hand-
crafted
codec can
always beat
a generic
serializer


A code-
generated
codec can
always beat
a hand-
crafted one


iMatix GSL:
technology
so dangerous
we had to
lock it up for
years


File transfer
is the
zombie
problem of
distributed
applications


Router
sockets are
the beating
heart of
every real
ØMQ
protocol
engine


The world
needs a
chunked,
flow-controlled,
restartable,
cancellable,
async,
multicast
file transfer
ØMQ protocol


No matter C: fetch
how hard S: chunk 1
you push, a S: chunk 2
file will not S: chunk 3
just go down
a socket


That C: fetch chunk 1
annoying S: send chunk 1
pause after C: fetch chunk 2
S: send chunk 2
you finish
C: fetch chunk 3
your beer, S: send chunk 3
before you C: fetch chunk 4
catch the
waiter's eye


You can, C: fetch chunk 1
and I've C: fetch chunk 2
tested this, C: fetch chunk 3
S: send chunk 1
order a new
C: fetch chunk 4
beer before S: send chunk 2
your old one S: send chunk 3
is empty


Request- C: subscribe
reply is just C: send credit
a vulgar S: send chunk
S: send chunk
subclass of
C: send credit
publish- S: send chunk
subscribe


On a router
socket, you
should
never hit the
high-water
mark


Heartbeats
are our
protocol's
way of
asking if we
still care


Protocol command_t
stack *request =
= command_decode
(socket)
message
codec execute_engine
+ (command)
protocol
engine


State
machines
are a perfect
domain
language for
protocol
engines


State
machines
can be cudly
and gentle,
when you
get to know
them


You don't
want to bet
against a
compiler


If you're not
thinking of
security,
security is
probably
thinking of
you


For
connected
bidirectional
protocols
over ØMQ
use SASL


For loosely
connected
and one-way
protocols
over ØMQ,
use AES and
such


secure-nom = open-peering
SASL over *use-peering

ØMQ is open-peering =
C:OHAI
darned *( S:ORLY C:YARLY )
simple ( S:OHAI-OK / S:WTF )

ORLY = 1*mechanism
challenge
mechanism = string
challenge = *OCTET

YARLY = mechanism
response
response = *OCTET


Theory is
fine in
theory, but
in practice,
practice is
better


FileMQ is a
file sharing
protocol and
stack over
ØMQ.

Reusable
until 2062


1. Aim for 50 years
To sum it up: 2. It's all about people
3. Minimal plausible solutions
zero.mq/ch6 4. To real immediate problems
5. Document the contracts
6. Cheap and Nasty codecs
The Weird 7. Code generation rocks
Fish Book, 8. Router sockets rock
coming soon 9. CBFC > HWM
from O'Reilly 10. Learn state machines
11. Learn about SASL

12. Worked example: FileMQ

Software Architecture over ZeroMQ

Recommended

More Related Content

Viewers also liked (20)

Recently uploaded (20)

Software Architecture over ZeroMQ