Architecting a next generation data platform

Hadoop Application
Architectures:
Architecting a Next
Generation Data Platform
Strata Data Conference, New York 2017
tiny.cloudera.com/app-arch-newyork
tiny.cloudera.com/nyquestions
Mark Grover | @mark_grover
Jonathan Seidman | @jseidman
Gwen Shapira | @gwenshap

Questions?
Logistics
▪ Break at 3:00 – 3:30 PM
▪ Questions at the end of each section
▪ Slides at tiny.cloudera.com/app-arch-newyork
▪ Code at https://github.com/hadooparchitecturebook/Taxi360

Questions?
About the book
▪ @hadooparchbook
▪ hadooparchitecturebook.com
▪ github.com/hadooparchitecturebook
▪ slideshare.com/hadooparchbook

Questions?
About the presenters
▪ Product Manager at Lyft
▪ Formerly Software Engineer
on Spark at Cloudera
▪ Committer on Apache Bigtop,
PMC member on Apache
Sentry, Apache Spot
(incubating)
▪ Contributor to Apache Spark,
Hadoop, Hive, Sqoop, Pig,
Flume
Mark Grover

Questions?
▪ Product Manager at Confluent
▪ PMC for Apache Kafka.
▪ Previously software engineer
at Cloudera
▪ @gwenshap on twitter
Gwen Shapira

Questions?
▪ Software Engineer at
Cloudera
▪ Contributor to Apache Sqoop.
▪ Previously Technical Lead on
the big data team at Orbitz,
co-founder of the Chicago
Hadoop User Group and
Chicago Big Data
Jonathan Seidman

Case Study Overview
Internet of Things and Entity 360

Questions?
Customer 360

Questions?
Connected Cars

Questions?
Entity (Taxi) 360 View
Geo-location/
Traffic Data
Customer Data
Maintenance
Data
Other Data
Sources
Streaming
Vehicle Data

Questions?
What Makes Hadoop a Fit?
Data Sources Extract Transform Load
The early days…

Questions?
What Makes Hadoop a Fit?
SERVERS MARTS EDWS DOCUMENTS STORAGE SEARCH ARCHIVE
ERP, CRM, RDBMS, MACHINES FILES, IMAGES, VIDEOS, LOGS, CLICKSTREAMS EXTERNAL DATA SOURCES
Today…

Questions?
Enabling a Range of New Use Cases…
Fraud Detection Market
Transactions
Internet of Things Network Security

Questions?
Hadoop Challenges
Kafka StreamsKafka Connect
Kafka

Questions?
Challenges – Architectural Considerations
▪ Reliable and scalable ingress of multiple data types and sources:
- High volume event data? Batch data?
▪ Reliable and scalable storage to support multiple workloads and access patterns
- Historical data? Real-time search? Analytics?
▪ Processing engines (for background processing):
- Stream processing? Batch processing?
▪ Data Modeling
- Modeling data for real-time random access? Analytic access? Batch access?

Case Study
Requirements
Overview

Questions?
Requirements
▪ Allow users (technical and non-technical) to analyze and visualize data…

Questions?
Requirements
▪ Provide analysts with query capabilities via a standard interface…

Questions?
Requirements
▪ Provide developers the ability to perform batch processing on historical data…

Questions?
Requirements
▪ To support all this, we need:
- Reliable ingestion of streaming and batch data.
- Ability to perform transformations on streaming data in flight.
- Ability to perform sophisticated processing of historical data.

High level architecture
Walkthrough

Questions?
Source Transport Stream
Processing
Storage Access
Data Producers Pub-Sub
Processing &
Ingestion Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT REST
NRT Dashboard

Questions?
TransportSource Stream
Processing
Storage Access
Data Producers
Processing &
Ingestion Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT REST
NRT Dashboard
Pub Sub

Questions?
Key to Customer 360 Success
Your project is only as good as the quality and variety of data sources
Geo-location/
Traffic Data
Customer DataMaintenance
Data
Other Data
Sources
Streaming
Vehicle Data
Files
CSV? XML?
JSON?
Twitter?
Mainframe?
Database Salesforce?
MQTT

Questions?
Data Producers: Flume vs. Kafka
▪ Flume – well integrated with Hadoop.
▪ Part of Hadoop ecosystem
▪ Great choice when ingesting data into HDFS.
▪ Can support simple transformations.
▪ Minimal coding – built in support for common data sources.
▪ Kafka – flexible, get-everything pipe
▪ Producers in ~ 20 languages
▪ REST API
▪ Huge connector ecosystem

Questions?
Kafka Clients
Apache Kafka Clients Ecosystem Clients

Questions?
REST Proxy
Talking to Non-native Kafka Apps and Outside the Firewall
REST Proxy
Non-Java Applications
Native Kafka Java Applications
REST / HTTP
Simplifies administrative
actions
Simplifies message creation
and consumption
Provides a RESTful
interface to a Kafka
cluster

Questions?
Kafka Connect
Streaming Data Capture
JDBC
Logs
MQTT
RDBMS
Key/Value
HDFS
Kafka Connect API
Kafka
Connector
Connector
Connector
Connector
Connector
Connector
Sources Sinks
Fault tolerant
Manage hundreds of data
sources and sinks
Preserves data schema
Part of Apache Kafka
project
Includes simple
transformations

Questions?
Ecosystem of Connectors
Databases Datastore/File Store
Analytics Applications / Other

Questions?
How Connect Works?
Log
Connector
MQTT
Connector
REST API
Logs MQTT
Log Task Log Task
MQTT
Task
MQTT
Task

Questions?
Schema Registry
Elastic
Cassandra
HDFS
Example Consumers
Serializer
Source 1
Serializer
Source 2
!
Kafka Topic!
Schema Registry
Define the expected fields for each Kafka topic
Automatically handle schema changes (e.g. new fields)Prevent backwards incompatible changes
Get different data sources to talk the same language

Questions?
TransportSource Stream
Processing
Storage Access
Processing &
Ingestion Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT REST
NRT Dashboard
Pub Sub

Questions?
But wait!
What about batch data?

Questions?
Source Buffer Stream
Processing
Storage Access
Pub-Sub
Processing &
Ingestion Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT REST
NRT Dashboard
Data Producers

Questions?
Buffering Data
▪ What do we mean by “buffering” and why do we need it?
event,event,event,event,event,event…
This is bad!
▪ Network partitions happen
▪ Producers and Consumers
work at different rates
▪ Reliable storage is hard
Stream processing is hard
Lets do one at a time

Questions?
Buffering Data – Message Brokers
Publisher
Publisher
Publisher
Message
Queue
Subscriber
Subscriber
Subscriber

Questions?
Source Buffer Stream
Processing
Storage Access
Processing &
Ingestion Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT REST
NRT Dashboard

Questions?
What is Kafka?
▪ It’s like a message queue, right?
- Actually, it’s a “distributed commit log”
- Or “streaming data platform”
0 1 2 3 4 5 6 7 8
Data
Source
Data
Consumer
A
Data
Consumer
B

Questions?
Topics and Partitions
▪ Messages are organized into topics, and each topic is split into partitions.
- Each partition is an immutable, time-sequenced log of messages on disk.
- Note that time ordering is guaranteed within, but not across, partitions.
0 1 2 3 4 5 6 7 8
0 1 2 3 4 5 6 7 8
0 1 2 3 4 5 6 7 8
Partition 0
Partition 1
Partition 2
Data
Source
Topic

Questions?
Consumers
In Our Architecture
Taxi Trip Data
Producer
Kafka
taxi-trip-input
Topic
Stream
Processing
(Analytic)
Stream
Processing
(Lookup)
Stream
Processing
(Search)
Stream
Processing
(Long Term)

Questions?
Input Events
CMT,2009-01-05 08:31:55,2009-01-05 8:37:50,1,0.90000000000000002,-73.977936999999997,
40.745919000000001,,,-
73.983609000000001,40.755051000000002,Credit,5.2999999999999998,0,,0.79000000000000004,0,6.0
899999999999999
vendor_name,Trip_Pickup_DateTime,Trip_Dropoff_DateTime,Passenger_Count,Trip_Distance,
Start_Lon,Start_Lat,Rate_Code,store_and_forward,End_Lon,End_Lat,Payment_Type,Fare_Amt,
surcharge,mta_tax,Tip_Amt,Tolls_Amt,Total_Amt

Questions?
Kafka Considerations – Reliability
▪ But remember there are tradeoffs…

Questions?
Kafka Considerations – Reliability
▪ Different reliability levels for topics:
Taxi Trip Data
Kafka
taxi-trip-input
Twitter customer-sentiment
100% – dups
are ok
(“At least
once”)
<=100%
(“At most
once”)
News Flash:
Kafka’s Exactly Once
Producer is on the way

Questions?
Kafka Reliability – Replication
Producer
Broker
Partition1
Partition2
Partition3
Leader

Questions?
Producer
Broker
Partition1
Partition2
Partition3

Questions?
Producer
Broker
Partition1
Partition2
Partition3
Broker
Partition1
Partition2
Partition3
Leader

Questions?
Kafka Reliability– Replication
Producer
Broker
Partition1
Partition2
Partition3
Broker
Partition1
Partition2
Partition3
Leader
Leader

Questions?
Producer
Broker
Partition1
Partition2
Partition3
Broker
Partition1
Partition2
Partition3
Broker
Partition1
Partition2
Partition3
Leader

Questions?
▪ So how does this relate to our application?
kafka-topics --zookeeper ZKHOST:ZKPORT –partition 2 --replication-factor 3
--create --topic taxi-trip-input
kafka-topics --zookeeper ZKHOST:ZKPORT –partition 2 --replication-factor 1
--create –topic customer-sentiment

Questions?
Kafka Reliability – Producers
Taxi Trip Data
Kafka
taxi_trip_input
Partition 1
Partition 2
Partition 3
Topic B
Partition 1
Partition 2
Partition 3
Message
failure?
Producer
Resend
message
acks=all

Questions?
Kafka Reliability – Producers
▪ What about duplicates?
Taxi Trip Data
Kafka
taxi_trip_input
Partition 1
Partition 2
Partition 3
Topic B
Partition 1
Partition 2
Partition 3
Producer
ID Message
1000 2009-01-04 03:02:00,1,2.629,...
1001 2009-01-04 03:38:00,3,4.549…
1001 2009-01-04 03:38:00,3,4.549…

Questions?
Kafka Scaling – Partitions
Producer
Kafka
taxi-trip-input
Partition 1
Partition 2
Partition 3
Consumer Group
Consumer
Consumer
Consumer

Questions?
Kafka Scaling – Partitions
Producer
Kafka
taxi-trip-input
Partition 1
Partition 2
Partition 3
Consumer Group
Consumer
Consumer
Consumer
Partition 4
Partition 5
Consumer
Consumer
Higher
throughput
Higher
throughput
More
resources
(memory)
More
resources
(file handles)
Producer

Questions?
How many partitions?
§ Adding partitions late in the game is painful
§ Basic formula:
total desired throughput / throughput of slowest consumer or producer
§ Or ~25GB disk space
§ Not too many because:
- Each partition takes broker heap memory and file handles
- Each partition slows down node shutdown / recovery
- 1000 – 4000 partitions per broker max
- Producers will produce smaller batches – lower throughput

Questions?
Kafka Scaling – Producers
Producer
Kafka
taxi-trip-input
Partition 1
Partition 2
Partition 3
Consumer Group
Consumer
Consumer
Consumer
Partition 4
Partition 5
Consumer
Consumer
Producer

Questions?
Guarding Against Message Loss
§ Producer – What happens if the producer loses connection to Kafka and the buffer overflows?
- You get an exception. You can choose to… block? Write to file?
§ Source – What happens if events are lost before getting sent to producer?
- Once again use some kind of buffer to provide sufficient retention of data.

Stream Processing
Considerations

Questions?
Processing
Storage Access
Custom
Producer
or
Processing &
Ingestion Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT REST
NRT Dashboard

Questions?
Streaming agenda
▪ What do we mean by streaming?
▪ Streaming use-cases
▪ Streaming semantics
▪ Which streaming engine to choose?
▪ Streaming in our use-case

Questions?
What do we mean by streaming?
Constant low
milliseconds & under
Low milliseconds to
seconds, delay in
case of failures
10s of seconds or
more, re-run in case
of failures
Real-time Near real-time Batch

Questions?
But, there’s no free lunch
Constant low
milliseconds & under
Low milliseconds to
seconds, delay in
case of failures
10s of seconds or
more, re-run in case
of failures
Real-time Near real-time Batch
“Difficult” architectures, lower
latency
“Easier” architectures, higher
latency

Questions?
Streaming Use-cases
▪ Ingestion (most relevant in our use-case)
▪ Simple transformations
- Decision (e.g. anomaly detection)
- Enrichment (e.g. add a state based on zipcode)
▪ Advanced usage
- Machine Learning
- Windowing

Questions?
#1 - Simple ingestion
Buffer
Event e Stream
Processing Long term
storage
Event e

Questions?
#2 - Enrichment
Buffer
Event e Stream
Processing Storage
Event e’
e’ = enriched event e
Context store

Questions?
#2 - Decision
Buffer
Event e Stream
Processing Storage
Event e’
e’ = e + decision
Rules

Questions?
#3 – Advanced usage
Buffer
Event e Stream
Processing Storage
Event e’
e’ = aggregation or
windowed aggregation
Model

Questions?
#1 – Simple Ingestion
1. Zero transformation
- No transformation, plain ingest
- Keep the original format – SequenceFile, Text, etc.
- Allows to store data that may have errors in the schema
2. Format transformation
- Simply change the format of the field
- To a structured format, say, Avro, for example
- Can do schema validation
3. Atomic transformation
- Mask a credit card number

Questions?
#2 - Enrichment
Buffer
Event e Stream
Processing Storage
Event e’
e’ = enriched event e
Context store
Need to store the
context
somewhere

Questions?
Where to store the context?
1. Locally Broadcast Cached Dim Data
- Local to Process (On Heap, Off Heap)
- Local to Node (Off Process)
2. Partitioned Cache
- Shuffle to move new data to partitioned cache
3. External Fetch Data (e.g. HBase, Memcached)

Questions?
#1a - Locally broadcast cached data
Could be
On heap or Off heap

Questions?
#1b - Off process cached data
Data is cached on the
node, outside of
process. Potentially in
an external system like
Rocks DB

Questions?
#2 - Partitioned cache data
Data is partitioned
based on field(s) and
then cached

Questions?
#3 - External fetch
Data fetched from
external system

Questions?
Partitioned cache + external

Questions?
Delivery Types
▪ At most once
- Not good for many cases
- Only where performance/SLA is more important than accuracy
▪ Exactly once
- Expensive to achieve but desirable
▪ At least once
- Easiest to achieve

Questions?
Semantics of our architecture
Source System 1
Destination
systemSource System 2
Source System 3
Ingest Extract Streaming
engine
Push
Message broker

Questions?
Classification of storage systems
▪ File based
- S3
- HDFS
▪ NoSQL
- HBase
- Cassandra
▪ Document based
- Search
▪ NoSQL-SQL
- Kudu

Questions?
Classification of storage systems
▪ File based
- S3
- HDFS
▪ NoSQL
- HBase
- Cassandra
▪ Document based
- Search
▪ NoSQL-SQL
- Kudu
De-duplication at file level
Semantics at key/record level

Which streaming
engine to choose?

Questions?
Processing
Storage Access
Processing &
Ingestion Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT REST
NRT Dashboard
Apache
Beam
Kafka
Streams

Questions?
Requirements
§Fault-tolerant and distributed
▪ Effectively once semantics
▪ Handle processing time vs. event time
▪ Allow stateful transformations

Questions?
Spark Streaming
▪ Micro batch based architecture
▪ Allows stateful transformations
▪ Feature rich
- Windowing
- Sessionization
- ML
- SQL (Structured Streaming)

Questions?
DStream
DStream
DStream
Single Pass
Source Receiver RDD
Source Receiver RDD
RDD
Filter Count Print
Source Receiver RDD
RDD
RDD
Single Pass
Filter Count Print
First
Batc
h
Second
Batch

Questions?
DStream
DStream
DStream
Single Pass
Source Receiver RDD
Source Receiver RDD
RDD
Filter Count
Print
Source Receiver
RDD
partitions
RDD
Parition
RDD
Single Pass
Filter Count
Pre-first
Batch
First
Batc
h
Second
Batch
Stateful
RDD 1
Print
Stateful
RDD 2
Stateful
RDD 1

Questions?
Spark Streaming - Gaps
§Not as low of a latency
- Efforts towards reducing latency e.g. RISElab’s Drizzle
§Global consistent execution state
- Stop overall execution of distributed computation
- Eagerly persist records in transit meaning larger snapshots

Questions?
Flink
▪ True “streaming” system, but not as feature rich as Spark
▪ Much better event time handling
▪ Good built-in backpressure support
▪ Allows stateful transformations
▪ Lower Latency
- No Micro Batching
- Asynchronous Barrier Snapshotting (ABS)

Questions?
Flink - ABS
Operator
Buffer

Questions?
Operator
Buffer
Operator
Buffer
Flink - ABS
Barrier 1A Hit
Barrier 1B
Still Behind

Questions?
Operator
Buffer
Flink - ABS
Both Barriers
Hit
Operator
Buffer
Barrier 1A Hit
Barrier 1B
Still Behind

Questions?
Operator
Buffer
Flink - ABS Both Barriers
Hit
Operator
Buffer Barrier is
combined and
can move on
Buffer can be
flushed out

Questions?
Storm
▪ Old school
▪ Didn’t manage state – had to use Trident
▪ No good support for batch processing

Questions?
Samza
▪ Good integration with Kafka
▪ Doesn’t support batch
▪ Forked by Kafka Streams

Questions?
Flume
▪ Well integrated with the Hadoop ecosystem
▪ Allowed interceptors (for simple transformations)
▪ Supports buffering
- Memory
- File
- Kafka
▪ But no real fault-tolerance
▪ No state management

Questions?
Kafka Streams
▪ Good integration with Kafka
▪ Light-weight library (not a framework)
▪ No micro-batching, uses Kafka as internal messaging layer
▪ Maintains local state per node (in RocksDB, or in memory
hash map)
▪ Handles late events
▪ Stream-to-stream joins

Questions?
Topic
Partition 1
Partition 2
Task 1 Re-partition topic
Partition 1
Partition 2
Task 3
Task 2
Task 4
Kafka Streams architecture

Questions?
Apache Beam
§ Abstraction on top of Streaming Engines
§ Best support for Google Dataflow

Questions?
Others
§ Apache Apex
§ Heron

Questions?
Spark Streaming
▪ We chose Spark Streaming because:
- Same execution engine for batch and streaming
- Similar code for batch and streaming
- Support for security, Kafka integration
- Thriving community

Questions?
Processing
Storage Access
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT REST
NRT Dashboard

Questions?
Structured Landing Zones
Relational
Nested
Time Series
Reversed Indexed
Traditional SQL
Optimized for nested Structures like JSON
Optimized Entity 360 and time base access
Optimized faceted charts and reverse index look ups
Graph Optimized for node and edges
Special Optimized for special use cases

Questions?
Structured Landing Zones
Relational
Nested
Time Series
Reverse Indexed
Traditional SQL
Optimized for nested structures like JSON
Optimized Entity 360 and time base access
Optimized faceted charts and reverse index look ups
Graph Optimized for node and edges

Questions?
Compression Styles and Entropy
Columns
Rows

Questions?
Compression Styles and Entropy
Block
Block
Block
Column
Column
Column
Column
Column
Column
Row
Group
Row
Group

Questions?
Compression Codecs
- Snappy: 2x-3x : Fast Read, Fast Write
- Lzo : 2x-3x : Fast Read, Fast Write
- Gzip : ~8x: ~Fast Read, Normal Write
- Default : ~8x: ~Fast Read, Normal Write
- BZip2 : ~10x ~Fast Read, Slow Write
- Others ..
- Always be skeptical
- All data compresses differently
- Use your own data

Questions?
Introducing the Hive Metastore
- Hive Metastore
- Adds a table like metadata layer over a file system, block store, NoSql, or other
- Allows for SQL access
- Allows for greater security options
- Allows for external metadata
- Allows for partitioning

Questions?
Typical Hive Table
- ParentFolder
- TableFolder
- Date=20171212
- DataFiles
- DataFiles
- Date=20171211
- DataFiles
- DataFiles

Questions?
Access Patterns
- Partitioning
- Filter push down
- Indexing should be considered poor
- Ideal for large scans

Questions?
Thinking about Object/Tables
1. Lets start off easy
1. Use Case: We are a Netflix type company and we have a log of users and movies watched
that looks something like this:
User ID Age Account Start
Date
Category Of User Movie Watched Movie Category Start Time Events List
Bob 42 12/12/2012 Basic Die Hard Action 5/4/2016 12:00 Play 0, pause at
15, FF at 40 to 55,
E at 90
Kat 31 12/12/2012 Platum Beauty and the
Beast
Family 5/4/2016 12:00 Play 0, pause at
15, FF at 40 to 55,
E at 90

Questions?
Thinking about Object/Tables
1. To make this into objects we need to do some separation
User
User_id
Age
St_dt
Category
Movie
Movie_id
Title
Category
Watch_session
Watch_id
St_dt
En_dt
User_id
Movie_id
Watch_Events
Watch_id
St_dt
Type
Duration
Category_Typ
Category_id
Stream_rt
Is_feature_enabled
1 *
*
1
1
*
1*

Questions?
Query Considerations
- Data is normally big so
- Partition respectively to access patterns
- Join with care
- Consider sampling or local testing before experimenting
- Data is files
- Latency to accessibility it high – seconds, minutes or more.

Questions?
Look for big tables
User
User_id
Age
St_dt
Category
Movie
Movie_id
Title
Category
Watch_session
Watch_id
St_dt
En_dt
User_id
Movie_id
Watch_Events
Watch_id
St_dt
Type
Duration
Category_Typ
Category_id
Stream_rt
Is_feature_enabled
1 *
*
1
1
*
1*

Questions?
Mutation Patterns
- File is written once and can not be mutated
- Fine for append or snapshot use cases
- Mutation will require a compaction

Questions?
Compaction Recap
Key Time Value
A 1 101
B 1 101
C 1 101
D 1 101
E 1 101
F 1 101
G 1 101
Key Time Value
A 2 102
D 2 102
F 2 102
F 3 103
H 3 103
Key Time Value
A 2 102
B 1 101
C 1 101
D 2 102
E 1 101
F 3 103
G 1 101
H 3 103

Questions?
View Strategies
Hive Relational Model
Hive Nested Model
Models
Hive Normal Views
Hive Materialized Table
Views
Use in the cases where the view requires
a join that is done through a shuffle
Use only for tables that filter
records/columns or use for marking fields

Questions?
Relational Storage Options

Questions?
Kudu

Questions?
Kudu Use Cases
§ Among other things, NRT availability of streaming data.
- A good fit for our application.
§ Also things like machine learning, time series, etc.

Questions?
Kudu In Our Architecture
def sendEntityToKudu(taxiEntityTableName: String, it: Iterator[(String, NyTaxiYellowEntityStateWrapper)],
kuduClient: KuduClient): Unit = {
val table = kuduClient.openTable(taxiEntityTableName)
val session = kuduClient.newSession()
session.setFlushMode(FlushMode.AUTO_FLUSH_BACKGROUND)
it.foreach(r => {
val state = r._2.state
val entity = r._2.entity
val operation: Operation = if (state.equals("New")) {
table.newInsert()
} else if (state.equals("Modified")) {
table.newUpdate()
} else {
null
}
...
https://github.com/hadooparchitecturebook/Taxi360/blob/master/src/main/scala/com/hadooparchitecturebook/taxi360/streaming/ingestion/kudu/SparkStreamingTaxiTripToKudu.scala

Questions?
Kudu In Our Architecture
…
sqlContext.read.options(kuduOptions).format("org.apache.kudu.spark.kudu").load.
registerTempTable("ny_taxi_trip_tmp")
//Vector
val vectorRDD:RDD[Vector] = sqlContext.sql("select * from ny_taxi_trip_tmp").map(r
=> {
val taxiTrip = NyTaxiYellowTripBuilder.build(r)
generateVectorOnly(taxiTrip)
})
println("--Running KMeans")
val clusters = KMeans.train(vectorRDD, numOfCenters, numOfIterations)
println(" > vector centers:")
clusters.clusterCenters.foreach(v => println(" >> " + v))
...
https://github.com/hadooparchitecturebook/Taxi360/blob/master/src/main/scala/com/hadooparchitecturebook/taxi360/etl/machinelearning/kudu/MlLibOnKudu.scala

Questions?
Nested
▪ Less Space than Denormalization
▪ Still have tables but the cost of joins is all but gone
▪ Also great for cartesian joins
- N x M vs N + M
▪ Not really supported yet with Kudu or HBase with SQL

Questions?
Nested Example
CREATE TABLE fact_contacts (id BIGINT, name STRING, address
STRING) STORED AS PARQUET;
CREATE TABLE dim_phones
(
contact_id BIGINT
, category STRING
, international_code STRING
, area_code STRING
, exchange STRING
, extension STRING
, mobile BOOLEAN
, carrier STRING
, current BOOLEAN
, service_start_date TIMESTAMP
, service_end_date TIMESTAMP
)

Questions?
Nested Example
CREATE TABLE contacts_detailed_phones
(
id BIGINT, name STRING, address STRING
, phone ARRAY < STRUCT <
category: STRING
, international_code: STRING
, area_code: STRING
, exchange: STRING
, extension: STRING
, mobile: BOOLEAN
, carrier: STRING
, current: BOOLEAN
, service_start_date: TIMESTAMP
, service_end_date: TIMESTAMP
>>
) STORED AS PARQUET;
https://www.cloudera.com/documentation/enterprise/latest/topics/impala_complex_types.html

Questions?
De-normalized vs Nested
- Nested Pros
- Co-location
- Faster to group by
- Faster to window
- Joins are free
- Less data
- Better compression
- Tables and Columns can be read with out penalty from one not read
- Great for limiting the effort are Cartesian Joins
- Nested Cons
- Size limitation of parent row
- Adding child requires the re-write the the whole parent record

Questions?
Options for appending Nested
- It is all about the parent record
- We can add more then one Partition key for the parent
- In our use case
- User & watch month or day

Questions?
Storage and In Memory
- Also don t limit the idea of nested to just tables
- In Spark they can be used as in memory constructs to
- conserve on networking
- In memory cost

Questions?
Nested Writing Example in Spark
{
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" },
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5007", "type": "Powdered Sugar" },
{ "id": "5006", "type": "Chocolate with Sprinkles" }
]

Questions?
Nested Writing Example in Spark
val jsonDF = hiveContext.read.json(jsonRDD)
jsonDF.write.parquet("./parquet")
hiveContext.createExternalTable("jsonNestedTable", "./parquet")

Questions?
Nested In Our Architecture
…
hiveContext.sql("create table " + hdfsTaxiNestedTableName + "( " +
" vender_id string," +
" trip array<struct< " +
" passenger_count: INT," +
" payment_type: STRING, " +
" total_amount: DOUBLE, " +
" fare_amount: DOUBLE " +
" >>" +
" ) stored as parquet")
val emptyDf = hiveContext.sql("select * from " + hdfsTaxiNestedTableName + " limit 0")
hiveContext.createDataFrame(newNestedDf, emptyDf.schema).registerTempTable("tmpNested")
hiveContext.sql("insert into " + hdfsTaxiNestedTableName + " select * from tmpNested")
…
https://github.com/hadooparchitecturebook/Taxi360/blob/master/src/main/scala/com/hadooparchitecturebook/taxi360/sql/kudu/KuduToNestedHDFS.scala

Questions?
Time Series Options
§ HBase and Cassandra

Questions?
Entity Centric Time Series
▪ Partition by Entity ID
▪ Order by Time
▪ Allows for free windowing
▪ Allows for fetching of single time window of single entity at web scale

Questions?
HBase Entity Time Series
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40
Cust-F, 20
Cust-F, 30
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40

Questions?
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40
Cust-F, 20
Cust-F, 30
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Rest Call Short Scan

Questions?
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40
Cust-F, 20
Cust-F, 30
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Mapper Mapper Mapper

Questions?
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40
Cust-F, 20
Cust-F, 30
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Mapper
Mapper Mapper

Questions?
What is meant by Bucketing and Sorting
- Partitioning on a Key
- Then sorting on that key + another field(s)
- Example
- User_id + Watch Event Time

Questions?
Example of Bucketed Sorted
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-F, 10
Cust-F, 20
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40

Questions?
Good for Appending Nested
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-F, 10
Cust-F, 20
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Cust-A, 50
Cust-A, 60
Cust-B, 50
Cust-B, 60
Cust-C, 50
Cust-D, 50
Cust-G, 50
Existing DataNew Data

Questions?
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-F, 10
Cust-F, 20
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Cust-A, 50
Cust-A, 60
Cust-B, 50
Cust-B, 60
Cust-C, 50
Cust-D, 50
Cust-G, 50
Shuffle Join

Questions?
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-F, 10
Cust-F, 20
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Cust-B, 50
Cust-B, 60
Cust-A, 50
Cust-A, 60
Cust-C, 50
Cust-D, 50
Cust-G, 50
Merge Join

Questions?
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-A, 50
Cust-A, 60
Cust-C, 50
Merge Join
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-A, 50
Cust-A, 60
Cust-C, 50
Order
Retained

Questions?
What else could be use Bucketing and Sorting For
- Windowing
- Point retrieval

Questions?
Bucketed & Sorted for Windowing
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-F, 10
Cust-F, 20
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Spark Mapper Spark Mapper Spark Mapper

Questions?
Bucketed Sorted in a NoSQL
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-F, 10
Cust-F, 20
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Rest Call Short Scan

Questions?
NoSQL
- Columnar

Questions?
What is a NoSQL
- It s not NO SQL
- It s not a Database
- Think of it more like a
- HashMap
- Log
- Bucketed and Ordered

Questions?
Hash Map
- There is a Key and a Value
- It is really fast to grab a key/value
- It is really fast to add a key/value
- Iteration is also possible
Key Value
A 1
B 1
C 1
D 1
E 1
F 1
G 1
Client

Questions?
Log with Compactions
- When new records come in they don t rewrite the old
- They compact in
Key Time Value
A 1 101
B 1 101
C 1 101
D 1 101
E 1 101
F 1 101
G 1 101
Key Time Value
A 2 102
D 2 102
F 2 102
F 3 103
H 3 103
Key Time Value
A 2 102
B 1 101
C 1 101
D 2 102
E 1 101
F 3 103
G 1 101
H 3 103

Questions?
HDFS
Log with Compactions
- Write Path
- Get Local for Record (Cached)
- First to WAL
- Then to Memstore
- Sorting & batching
- Flush to New Hfile
- Later Hfiles will be compacted
Client
Master
RegionServer
Memstore
HFiles New HFiles
HFiles
WAL

Questions?
HDFS
Ordered
- All Records Columns are ordered
- Ordering allows for simpler indexing
- Ordering allows for simpler compactions
- We will also use this ordering
- Windowing
- Time series
- Local scanning
Client
Master
RegionServer
Memstore
HFiles New HFiles
HFiles

Questions?
Bucketing or Partitions
- HBase
- Out of the Box:
- Range
- Desired:
- Salt
- Cassandra
- Out of the Box:
- HashMod
- Bucketed HashMod

Questions?
So what about SQL
- Well SQL could totally work
- CQL for cassandra
- Hive and SparkSQL on HBase
- Why is it not the best idea
- Built more for point look ups
- Scans are not as fast as parquet
- However the mutability may be more important than speed
- Partitioning is not simple
- It must be put into the key

Questions?
Let s talk about CAP for a Minute
- Strong Consistency
- HBase & Kudu
- Variable Consistency
- Cassandra

Questions?
HBase Model
Client
Master
Region Server 1
Region Server 2
- Region Server owns range splits
- Region Server 1 fails
- Master needs to figure that out
- Master needs to assign new Region Server to own splits
- Region Server 2 has to get organized
- Region Server 2 is read to server reads and writes

Questions?
Cassandra Model
Client
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Replica Node
(Random Node)
Client
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Replica Node
(Random Node)

Questions?
Cassandra Model
Client
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Client

Questions?
Cassandra Model (Common Models)
Client
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Client
Client
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Client
Client
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Replica Node
(Has Replica)
Client
3 Write - 1 Read
1 Write - 3 Read
1 Write - 1 Read

Questions?
NoSQL - Others
- Document
- Mongo
- CouchBase
- Spanner-Inspired
- Kudu
- CockroachDB
- Druid.IO

Questions?
NoSQL - Transitions
- Some have them
- Think about kafka

Questions?
Lucene Indexing (Features)
- We don t have enough time in this whole class
- Ordering logic
- NGrams
- Weights
- Text Indexing
- Translations
- Facets *

Questions?
Lucene Indexing (Facets)
- Facets are a side effect of out wonderful indexes
- It allows us to counts all the document that below to given indexes to produce
- Grouped Counts
- Charts and Graphs (kibana or Banana)
- People will also call this access pattern cubing a dataset

Questions?
Lucene Indexing (Kibana & Banana)

Questions?
Lucene Indexing (Facets Example)
- Time Series Example
Document
ID
Hour of Day User State Event
1 12 4201 MD click
2 12 4202 VA click
3 12 4203 VA click
4 1 4201 MD click
5 1 4202 VA view
6 2 4204 CA click
7 2 4205 VA view
8 2 4201 MD click

Questions?
Hour of
Day
12 1 2 3
1 4 5
2 6 7 8 9
Document
ID
Hour of
Day
User State Event
1 12 4201 MD click
2 12 4202 VA click
3 12 4203 VA click
4 1 4201 MD click
5 1 4202 VA view
6 2 4204 CA click
7 2 4205 VA view
8 2 4201 MD click
9 2 4204 CA click
User
4201 1 4 8
4202 2 5
4203 3
4204 6 9
4205 7
State
MD 1 4 8
VA 2 3 5 7
CA 6 9
Event
click 1 2 3 4 6 8 9
view 5 7

Questions?
- Events per hour
- Simple array count
Hour of
Day
12 1 2 3
1 4 5
2 6 7 8 9

Questions?
- Events per hour by State
- Simple array count
State
MD 1 4 8
VA 2 3 5 7
CA 6 9
Hour of
Day
12 1 2 3
1 4 5
2 6 7 8 9

Questions?
- Note the bucketing and ordered pattern
Hour of
Day 2
State
MD
State
VA
State CA
6 1 2 6
7 4 3 9
8 8 5
9 7State
MD 1 4 8
VA 2 3 5 7
CA 6 9
Hour of
Day
12 1 2 3
1 4 5
2 6 7 8 9

Questions?
Hour of
Day 2
State
MD
State
VA
State CA
6 1 2 6
7 4 3 9
8 8 5
9 7
Hour of
Day 2
State
MD
State
VA
State CA
6 1 2 6
7 4 3 9
8 8 5
9 7
+1 CA

Questions?
Hour of
Day 2
State
MD
State
VA
State CA
6 1 2 6
7 4 3 9
8 8 5
9 7
+1VA
Hour of
Day 2
State
MD
State
VA
State CA
6 1 2 6
7 4 3 9
8 8 5
9 7
+1 MD
Hour of
Day 2
State
MD
State
VA
State CA
6 1 2 6
7 4 3 9
8 8 5
9 7
+1 CA

Questions?
Partitioning
- SolR and Elastic Search partition the document o land on all nodes
- This means
- You have the power of the cluster when querying
- This mean you are accessing the cluster when querying

Questions?
Writing Latency
- Lucene Indexing is more expensive then NoSQL work
- Think of it as micro batching
- Larger batches ~= better throughput
- Compaction is also invalid
- Deletes impact storage and performance until they are compacted

Questions?
Storage Cost
- TTL is your friend
- Think of Lucene based systems as great if
- You dataset is manageable in size
- You have a good TTL strategy
- You have a boat load of money

Questions?
Thinking in terms of Graphs
- Nodes and Edges
Node:1
…
Node:3
…
Node:2
…
Node:0
…
Friend
Child
Father
CoachWife

Questions?
Thinking in terms of Graphs
- Use cases
- Querying
- Cassandra with Sparkle
- Neo4j
- Batch operations
- Giraph
- GraphX
- GraphLab

Questions?
BSP Bulk Synchronous Parallel
- Process every Node Atomically
- Node gets all messages sent to it
- Nodes can mutate them selves and their edges
- Nodes can send messages to other nodes
- But nothing is received yet
- BSP waits until all the Node processing is done
- Then send messages to the right partition
- Repeat

Questions?
Storage
Processing
Access

Batch Processing
Considerations

Questions?
Why have batch processing?
▪ When you need a larger context
- Say, to train a model
▪ Complex periodic job that does something
- Convert data to a nested structure for reduced number of shuffles
▪ In our use-case,
- Kudu -> HDFS Nested is batch processing
- KMeans calculation is also in bash

Questions?
Batch processing options
▪ Spark (+ MLlib)
▪ MapReduce (+ Mahout)
▪ Flink (+ Flink ML)

Questions?
Spark
▪ Pretty popular
▪ Much faster than MapReduce
▪ Thriving community

Questions?
MapReduce
▪ Sloooooow

Questions?
Flink
▪ Pretty popular
▪ Batch is a special case of Streaming
▪ Developing community

Questions?
In our use-case
▪ We chose Spark
- We were using Spark Streaming anyways
- Similar code between Spark and Spark Streaming
- Thriving community

Interactive
Data Access
Considerations

Questions?
Types of data access
▪ REST server/APIs for querying entities and aggregates
▪ UI for displaying search facets
▪ SQL engine

Questions?
Why have REST server?
▪ Tired of business people telling us how to access data
▪ Serves as an interface between the data engineers and business folks
▪ Lets business folks decide access patterns
▪ Engineers to optimize those patterns
▪ Brownie points from your boss
▪ And, it’s not that difficult to write!

Questions?
Don’t believe me?
import org.mortbay.jetty.Server
import org.mortbay.jetty.servlet.{Context, ServletHolder}
…
val server = new Server(port)
val sh = new ServletHolder(classOf[ServletContainer])
sh.setInitParameter("com.sun.jersey.config.property.resourceConfigClass",
"com.sun.jersey.api.core.PackagesResourceConfig")
sh.setInitParameter("com.sun.jersey.config.property.packages",
"com.hadooparchitecturebook.taxi360.server.hbase")
sh.setInitParameter("com.sun.jersey.api.json.POJOMappingFeature", "true”)
val context = new Context(server, "/", Context.SESSIONS)
context.addServlet(sh, "/*”)
server.start()
server.join()

Questions?
Then, write a ServiceLayer
@GET
@Path("vender/{venderId}/timeline")
@Produces(Array(MediaType.APPLICATION_JSON))
def getTripTimeLine (@PathParam("venderId") venderId:String,
@QueryParam("startTime") startTime:String = Long.MinValue.toString,
@QueryParam("endTime") endTime:String = Long.MaxValue.toString):
Array[NyTaxiYellowTrip] = {

Questions?
Use REST! Say no to business people!
▪ Access data like so:
http://<serverURL>:8080/vendor/{vendorId}/timeline

Questions?
UI requirements
Something that can
▪Represent search results really well
▪Integrates with Apache Solr on Hadoop

Questions?
UI options
▪ Hue
▪ Banana
▪ Kibana

Questions?
We choose Hue
▪ Because it’s included
▪ Please look at the others

Questions?
SQL engine criteria
▪ Low latency SQL access
▪ Allows for high concurrency
▪ JDBC/ODBC integration
▪ Capable of large scale aggregation
▪ Optionally integrates with Kudu for real-time updates to SQL tables

Questions?
Apache Hive
▪ Good JDBC integration
▪ Not really low latency, even when using Tez
▪ Doesn’t integrate with Kudu
§ Can run with MapReduce, Spark, or Tez

Questions?
Presto
▪ Low latency SQL engine from Facebook
▪ Provides JDBC/ODBC access
▪ Is only in-memory, large aggregations can lead to OOM errors

Questions?
Apache Impala
▪ Low latency SQL access
▪ Excellent concurrency support
▪ Integrates with Kudu for real-time SQL

Questions?
Apache Drill
▪ Similar in architecture to Impala

Questions?
Spark SQL
▪ Builds on top of Spark
▪ JDBC/ODBC access only via Spark Thrift Server
- Doesn’t scale well with larger number of concurrent users
- Doesn’t fully provide secure access.

Questions?
We choose
▪ Spark SQL
▪ Impala

Questions?
Processing
Storage Access
Processing &
Ingestion Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series Lookup
Batch
Processing
SQL
NRT Rest
NRT Dashboard

Questions?
Storage
Processing
Access
Batch
Processing
SQL
NRT REST
NRT Dashboard

Questions?
Access
Processing
Storage

Questions?
Processing
Storage Access

Questions?
Other Sessions
▪ Ask Us Anything session – Thursday, 1:15 PM
▪ The Three Realities of Modern Programming: the Cloud, Microservices, and the
Explosion of Data (Gwen) – Thursday 11:20 AM
▪ One Cluster Does Not Fit All: Architecture Patterns for Multicluster Apache Kafka
Deployments (Gwen) – Thursday 2:05 PM
▪ Managing Successful Big Data Projects (Ted Malaska and Jonathan) – Thursday
4:35 PM

Thank you!
@hadooparchbook
tiny.cloudera.com/app-arch-newyork
Mark Grover | @mark_grover
Gwen Shapira | @gwenshap
Jonathan Seidman | @jseidman

Architecting a next generation data platform

Recommended

More Related Content

What's hot (20)

Similar to Architecting a next generation data platform (20)

Recently uploaded (20)

Architecting a next generation data platform