A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne '14)

NoSQL Matter 2014 - A real-time (Lambda) Architecture using Hadoop & Storm - #nosql14
A real-time Lambda Architecture using Hadoop & Storm
NoSQL Matters Cologne 2014 by Nathan Bijnens

Speaker
Nathan Bijnens
Big Data Engineer @ Virdata
@nathan_gs

Computing Trends
Past
Computation (CPUs)
Expensive
Disk Storage Expensive
Coordination Easy
(Latches Don’t Often Hit)
DRAM Expensive
Computation Cheap
(Many Core Computers)
Disk Storage Cheap
(Cheap Commodity Disks)
Coordination Hard
(Latches Stall a Lot, etc)
DRAM / SSD
Getting Cheap
Current
Source: Immutability Changes Everything - Pat Helland, RICON2012

Credits
Nathan Marz
● Ex-Backtype & Twitter
● Startup in Stealthmode
Creator of
● Storm
● Cascalog
● ElephantDB
Coined the term Lambda Architecture.
manning.com/marz

a Data System

Not all information is equal.
Some information is derived from other pieces of information.
Data is more than Information

Eventually you will reach the most ‘raw’ form of
information.
This is the information you hold true, simply because it exists.
Let’s call this ‘data’, very similar to ‘event’.
Data is more than Information

Events used to manipulate the master data.
Events: Before

Today, events are the master data.
Events: After

Let’s store everything.
Data System

Data is Immutable.
Data System

Data is Time Based.
Data System

Capturing change
INSERT INTO contact (name, city) VALUES (‘Nathan’, ‘Antwerp’)
UPDATE contact SET city = ‘Cologne’ WHERE name = ‘Nathan’
Traditionally

Capturing change
INSERT INTO contact (name, city, timestamp) VALUES (‘Nathan’, ‘Antwerp’, 2008-10-11 20:00Z)
INSERT INTO contact (name, city, timestamp) VALUES (‘Nathan’, ‘Cologne’, 2014-04-29 10:00Z)
in a Data System

The data you query is often transformed, aggregated, ...
Rarely used in it’s original form.
Query

Query = function ( all data )
Query

Query: Number of people living in each city
Person City Timestamp
Nathan Antwerp 2008-10-11
John Cologne 2010-01-23
Dirk Antwerp 2012-09-12
Nathan Cologne 2014-04-29
City Count
Antwerp 1
Cologne 2

Query
All Data QueryPrecomputed
View

Layered Architecture
Batch Layer
Speed Layer
Serving
Layer

Layered Architecture
Hadoop ElephantDB
Incoming Data
Cassandra
Query

Batch Layer

Batch Layer
Hadoop ElephantDB
Incoming Data

Batch Layer
The batch layer can calculate anything, given enough time...
Unrestrained computation.

No need to De-Normalize.
The batch layer stores the data normalized, the generated views are often, if not always denormalized.
Batch Layer

Horizontally scalable.
Batch Layer

High Latency.
Let’s for now pretend the update latency doesn’t matter.
Batch Layer

Functional computation, based on immutable inputs, is
idempotent.
Batch Layer

Stores a master copy of the data set
Batch Layer
… append only

Batch: view generation
Master Dataset
View #1
View #3
View #2
MapReduce
MapReduce
MapReduce

MapReduce
1. Take a large data set and divide it into subsets
2. Perform the same function on all subsets
3. Combine the output from all subsets
…
…
Output
DoWork() DoWork() DoWork() …
MAPREDUCE

MapReduce

Serialization & Schema
Catch errors as quickly as they happen.
Validate on write vs on read.
Catch errors as quickly as they happen.
Validate on write vs on read.

CSV is actually a serialization language that is just
poorly defined.

Use a format with a schema
● Thrift
● Avro
● Protocolbuffers
Could be combined with Parquet.
Added bonus: it’s faster and uses less space.

Batch View Database
No random writes required.
Read Only database

Every iteration produces the views from scratch.
Batch View Database

Pure Lambda databases
● ElephantDB
● SploutSQL
Databases with a batch load & read only views
● Voldemort
Other databases that could be used
● ElasticSearch/Solr: generate the lucene indexes using MapReduce
● Cassandra: generate sstables
● ...
Batch View Databases

Batch Layer
Without the associated complexities.
Eventually consistent

Batch Layer
Data absorbed into Batch Views
Time
Now
We are not done yet…
Not yet absorbed.
Just a few hours of data.

Speed Layer

Speed Layer
Hadoop ElephantDB
Incoming Data
Cassandra

Stream processing.
Speed Layer

Continuous computation.
Speed Layer

Storing a limited window of data.
Compensating for the last few hours of data.
Speed Layer

All the complexity is isolated in the Speed Layer.
If anything goes wrong, it’s auto-corrected.
Speed Layer

You have a choice between:
● Availability
○ Queries are eventual consistent
● Consistency
○ Queries are consistent
CAP

Eventual accuracy
Some algorithms are hard to implement in real-time.
For those cases we could estimate the results.

Storm
Speed Layer

Message passing
Storm

Distributed processing
Storm

Horizontally scalable.
Storm

Incremental algorithms
Storm

Fast.
Storm

Storm

Storm
Tuple
Stream

Storm
Spout
Bolt

Storm
Grouping

Data Ingestion
Queues & Pub/Sub models are a natural fit.

● Kafka
● Flume
● Scribe
● *MQ
● …
Data Ingestion

Speed Layer Views
The views need to be stored in a random writable
database.

The logic behind a R/W database is much more
complex than a read-only view.
Speed Layer Views

The views are stored in a Read & Write database.
● Cassandra
● Hbase
● Redis
● SQL
● ElasticSearch
● ...
Speed Layer Views

Serving Layer

Serving Layer
Hadoop ElephantDB
Incoming Data
Cassandra
Query

Serving Layer
Random reads.

This layer queries the batch & real-time views and
merges it.
Serving layer

How to query an Average?
Serving Layer

Side note: CQRS

CQRS
Source: martinfowler.com/bliki/CQRS.html - Martin Fowler

CQRS & Event Sourcing
Event Sourcing
● Every command is a new event.
● The event store keeps all events, new events are
appended.
● Any query loops through all related events, even
to produce an aggregate.
source: CQRS Journey - Microsoft Patterns & Practices

Lambda Architecture

Lambda Architecture
The Lambda Architecture can discard any view, batch
and real-time, and just recreate everything from the
master data.

Mistakes are corrected via recomputation.
Write bad data? Remove the data & recompute.
Bug in view generation? Just recompute the view.
Lambda Architecture

Data storage is highly optimized.
Lambda Architecture

Immutability changes everything.
Lambda Architecture

Questions?
@nathan_gs #nosql14
nathan@nathan.gs / slideshare.net/nathan_gs
lambda-architecture.net / @LambdaArch / #LambdaArch

Virdata is the cross-industry cloud service/platform for the Internet
of Things. Designed to elastically scale to monitor and manage an
unprecedented amount of devices and applications using
concurrent persistent connections, Virdata opens the door to
numerous new business opportunities.
Virdata combines Publish-Subscribe based Distributed Messaging,
Complex Event Processing and state-of-the-art Big Data paradigms
to enable both historical & real-time monitoring and near real-time
analytics with a scale required for the Internet of Things.

Acknowledgements
I would like to thank Nathan Marz for writing a very insightful book, where most of the ideas in this presentation come from.
Parts of this presentation has been created while working for datacrunchers.eu, I thank them for the opportunities to speak about the
Lambda Architecture both at clients and at conferences. DataCrunchers is the first Big Data agency in Belgium.
Schema’s & Pictures:
Computing Trends: Immutability Changes Everything - Pat Helland, RICON2012
MapReduce #1: PolybasePass2012.pptx - David J. DeWitt, Microsoft Gray Systems Lab
MapReduce #2: Introduction to MapReduce and Hadoop - Shivnath Babu, Duke
CQRS: martinfowler.com/bliki/CQRS.html - Martin Fowler
CQRS & Event Sourcing: CQRS Journey - Adam Dymitruk, Josh Elster & Mark Seemann, Microsoft Patterns & Practices

Thank you
@nathan_gs
nathan@nathan.gs

A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne '14)

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne '14) (20)

More from Nathan Bijnens (13)

Recently uploaded (20)

A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne '14)