Architectural considerations for Hadoop Applications

Architectural
Considerations
for Hadoop
Applications
Strata+Hadoop World, San Jose – February 18th, 2015
tiny.cloudera.com/app-arch-slides
Mark Grover | @mark_grover
Ted Malaska | @TedMalaska
Jonathan Seidman | @jseidman
Gwen Shapira | @gwenshap

2
About the book
•  @hadooparchbook
•  hadooparchitecturebook.com
•  github.com/hadooparchitecturebook
•  slideshare.com/hadooparchbook
©2014 Cloudera, Inc. All Rights Reserved.

3
About the presenters
•  Principal Solutions
Architect at Cloudera
•  Previously, lead
architect at FINRA
•  Contributor to Apache
Hadoop, HBase,
Flume, Avro, Pig and
Spark
•  Senior Solutions
Architect/Partner
Enablement at
Cloudera
•  Previously, Technical
Lead on the big data
team at Orbitz
Worldwide
•  Co-founder of the
Chicago Hadoop User
Group and Chicago Big
Data
Ted Malaska Jonathan Seidman

4
About the presenters
•  Solutions Architect turned
Software Engineer at
Cloudera
•  Committer on Apache
Sqoop
Flume and Apache Kafka
•  Software Engineer at
Cloudera
•  Committer on Apache
Bigtop, PMC member on
Apache Sentry
(incubating)
Hadoop, Spark, Hive,
Sqoop, Pig and Flume
Gwen Shapira Mark Grover

5
Logistics
•  Break at 10:30-11:00 PM
•  Questions at the end of each section

6
Case Study
Clickstream Analysis

7
Analytics

8
Analytics

9
Analytics

10
Analytics

11
Analytics

12
Analytics

13
Analytics

14
Web Logs – Combined Log Format
244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0"
200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/
5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/36.0.1944.0 Safari/537.36”
244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?
productID=1023 HTTP/1.0" 200 3757 "http://www.casualcyclist.com"
"Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; HTC Vision Build/
GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile
Safari/533.1”

15
Clickstream Analytics
244.157.45.12 - - [17/Oct/
2014:21:08:30 ] "GET /seatposts
HTTP/1.0" 200 4463 "http://
bestcyclingreviews.com/
top_online_shops" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_9_2)
AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/36.0.1944.0 Safari/
537.36”

16
Similar use-cases
•  Sensors – heart, agriculture, etc.
•  Casinos – session of a person at a table

17
Pre-Hadoop
Architecture

18
Click Stream Analysis (Before Hadoop)
Web logs
(full fidelity)
(2 weeks)
Data Warehouse
Transform/
Aggregate Business
Intelligence
Tape Archive

19
Problems with Pre-Hadoop Architecture
•  Full fidelity data is stored for small amount of time (~weeks).
•  Older data is sent to tape, or even worse, deleted!
•  Inflexible workflow - think of all aggregations beforehand

20
Effects of Pre-Hadoop Architecture
•  Regenerating aggregates is expensive or worse, impossible
•  Can’t correct bugs in the workflow/aggregation logic
•  Can’t do experiments on existing data

21
Why is Hadoop A
Great Fit?

22
Why is Hadoop a great fit?
•  Volume of clickstream data is huge
•  Velocity at which it comes in is high
•  Variety of data is diverse - semi-structured data
•  Hadoop enables
–  active archival of data
–  Aggregation jobs
–  Querying the above aggregates or raw fidelity data

23
Click Stream Analysis (with Hadoop)
Web logs Hadoop Business
Intelligence
Active archive (no tape)
Aggregation engine
Querying engine

24
Challenges of Hadoop Implementation

25
Challenges of Hadoop Implementation

26
Other challenges - Architectural
Considerations
•  Storage managers?
–  HDFS? HBase?
•  Data storage and modeling:
–  File formats? Compression? Schema design?
•  Data movement
–  How do we actually get the data into Hadoop? How do we get it out?
•  Metadata
–  How do we manage data about the data?
•  Data access and processing
–  How will the data be accessed once in Hadoop? How can we transform it? How do
we query it?
•  Orchestration
–  How do we manage the workflow for all of this?

27
Case Study
Requirements
Overview of Requirements

28
Overview of Requirements
Data
Sources
Ingestion
Raw Data
Storage
(Formats,
Schema)
Processed
Data Storage
(Formats,
Schema)
Processing
Data
Consumption
Orchestration
(Scheduling,
Managing,
Monitoring)

29
Case Study
Requirements
Data Ingestion

30
Data Ingestion Requirements
Web Servers
Web Servers
Web Servers
Web Servers
Web Servers
Logs
244.157.45.12 - - [17/
Oct/2014:21:08:30 ]
"GET /seatposts HTTP/
1.0" 200 4463 …
CRM
Data
ODS
Web Servers
Web Servers
Web Servers
Web Servers
Web Servers
Logs
Hadoop

31
Data Ingestion Requirements
•  So we need to be able to support:
–  Reliable ingestion of large volumes of semi-structured event data arriving with
high velocity (e.g. logs).
–  Timeliness of data availability – data needs to be available for processing to
meet business service level agreements.
–  Periodic ingestion of data from relational data stores.

32
Case Study
Requirements
Data Storage

33
Data Storage Requirements
Store all the data
Make the data
accessible for
processing
Compress
the data

34
Case Study
Requirements
Data Processing

35
Processing requirements
Be able to answer questions like:
•  What is my website’s bounce rate?
–  i.e. how many % of visitors don’t go past the landing page?
•  Which marketing channels are leading to most sessions?
•  Do attribution analysis
–  Which channels are responsible for most conversions?

36
Sessionization
Website visit
Visitor 1
Session 1
Visitor 1
Session 2
Visitor 2
Session 1
> 30 minutes

37
Case Study
Requirements
Orchestration

38
Orchestration is simple
We just need to execute actions
One after another

39©2014 Cloudera, Inc. All Rights Reserved.
Actually,
we also need to handle errors
And user notifications
….

And…
•  Re-start workflows after errors
•  Reuse of actions in multiple workflows
•  Complex workflows with decision points
•  Trigger actions based on events
•  Tracking metadata
•  Integration with enterprise software
•  Data lifecycle
•  Data quality control
•  Reports

OK, maybe we need a product
To help us do all that

42
Architectural
Considerations
Data Modeling

43
Data Modeling Considerations
•  We need to consider the following in our architecture:
–  Storage layer – HDFS? HBase? Etc.
–  File system schemas – how will we lay out the data?
–  File formats – what storage formats to use for our data, both raw and
processed data?
–  Data compression formats?

44
Architectural
Considerations
Data Modeling – Storage Layer

45
Data Storage Layer Choices
•  Two likely choices for raw data:

46
Data Storage Layer Choices
•  Stores data directly as files
•  Fast scans
•  Poor random reads/writes
•  Stores data as Hfiles on
HDFS
•  Slow scans
•  Fast random reads/writes

47
Data Storage – Storage Manager
Considerations
•  Incoming raw data:
–  Processing requirements call for batch transformations across multiple
records – for example sessionization.
•  Processed data:
–  Access to processed data will be via things like analytical queries – again
requiring access to multiple records.
•  We choose HDFS
–  Processing needs in this case served better by fast scans.

48
Architectural
Considerations
Data Modeling – Raw Data Storage

49
Storage Formats – Raw Data and Processed
Data
Processed
Data
Raw
Data

50
Data Storage – Format Considerations
Click to enter confidentiality information
Logs
(plain
text)

51
Data Storage – Format Considerations
Logs
(plain
text)
Logs
(plain
text)
Logs
(plain
text)
Logs
(plain
text)
Logs
(plain
text)
Logs
(plain
text)
Logs
Logs
Logs
Logs
Logs

52
Data Storage – Compression
snappy
Well, maybe.
But not splittable.
X
Splittable. Getting
better…
Hmmm….Splittable, but no...

53
Raw Data Storage – More About Snappy
•  Designed at Google to provide high compression speeds with reasonable
compression.
•  Not the highest compression, but provides very good performance for
processing on Hadoop.
•  Snappy is not splittable though, which brings us to…

54
Hadoop File Types
•  Formats designed specifically to store and process data on Hadoop:
–  File based – SequenceFile
–  Serialization formats – Thrift, Protocol Buffers, Avro
–  Columnar formats – RCFile, ORC, Parquet

55
SequenceFile
•  Stores records as
binary key/value pairs.
•  SequenceFile “blocks”
can be compressed.
•  This enables
splittability with non-
splittable compression.

56
Avro
•  Kinda SequenceFile on
Steroids.
•  Self-documenting –
stores schema in
header.
•  Provides very efficient
storage.
•  Supports splittable
compression. ©2014 Cloudera, Inc. All Rights Reserved.

57
Our Format Recommendations for Raw
Data…
•  Avro with Snappy
–  Snappy provides optimized compression.
–  Avro provides compact storage, self-documenting files, and supports schema
evolution.
–  Avro also provides better failure handling than other choices.
•  SequenceFiles would also be a good choice, and are directly
supported by ingestion tools in the ecosystem.
–  But only supports Java.

58
But Note…
•  For simplicity, we’ll use plain text for raw data in our example.

59
Architectural
Considerations
Data Modeling – Processed Data Storage

60
Storage Formats – Raw Data and Processed
Data
Processed
Data
Raw
Data

61
Access to Processed Data
Column A Column B Column C Column D
Value Value Value Value
Analytical
Queries

62
Columnar Formats
•  Eliminates I/O for columns that are not part of a query.
•  Works well for queries that access a subset of columns.
•  Often provide better compression.
•  These add up to dramatically improved performance for many
queries.
1 2014-10-1
3
abc
2 2014-10-1
4
def
3 2014-10-1
5
ghi
1 2 3
2014-10-1
3
2014-10-1
4
2014-10-1
5
abc def ghi

63
Columnar Choices – RCFile
•  Designed to provide efficient processing for Hive queries.
•  Only supports Java.
•  No Avro support.
•  Limited compression support.
•  Sub-optimal performance compared to newer columnar formats.

64
Columnar Choices – ORC
•  A better RCFile.
•  Also designed to provide efficient processing of Hive queries.
•  Only supports Java.

65
Columnar Choices – Parquet
•  Designed to provide efficient processing across Hadoop
programming interfaces – MapReduce, Hive, Impala, Pig.
•  Multiple language support – Java, C++
•  Good object model support, including Avro.
•  Broad vendor support.
•  These features make Parquet a good choice for our processed data.

66
Architectural
Considerations
Data Modeling – Schema Design

67
HDFS Schema Design – One
Recommendation
/user/<username> - User specific data, jars, conf files
/etl – Data in various stages of ETL workflow
/tmp – temp data from tools or shared between users
/data – processed data to be shared data with the entire organization
/app – Everything but data: UDF jars, HQL files, Oozie workflows

68
Partitioning
•  Split the dataset into smaller consumable chunks.
•  Rudimentary form of “indexing”. Reduces I/O needed to process
queries.

69
Partitioning
dataset
col=val1/file.txt
col=val2/file.txt
…
col=valn/file.txt
dataset
file1.txt
file2.txt
…
filen.txt
Un-partitioned HDFS
directory structure
Partitioned HDFS
directory structure

70
Partitioning considerations
•  What column to partition by?
–  Don’t have too many partitions (<10,000)
–  Don’t have too many small files in the partitions
–  Good to have partition sizes at least ~1 GB, generally a multiple of block size.
•  We’ll partition by timestamp. This applies to both our raw and
processed data.

71
Partitioning For Our Case Study
•  Raw dataset:
–  /etl/BI/casualcyclist/clicks/rawlogs/year=2014/month=10/day=10!
•  Processed dataset:
–  /data/bikeshop/clickstream/year=2014/month=10/day=10!

72
Architectural
Considerations
Data Ingestion

73©2014 Cloudera, Inc. All rights reserved.
•  Omniture data on FTP
•  Apps
•  App Logs
•  RDBMS
Typical Clickstream data sources

Getting Files from FTP

curl ftp://myftpsite.com/sitecatalyst/
myreport_2014-10-05.tar.gz
--user name:password | hdfs -put - /etl/clickstream/raw/
sitecatalyst/myreport_2014-10-05.tar.gz
Don’t over-complicate things

Apache NiFi

Reliable, distributed and highly available systems
That allow streaming events to Hadoop
Event Streaming – Flume and Kafka

•  Many available data collection sources
•  Well integrated into Hadoop
•  Supports file transformations
•  Can implement complex topologies
•  Very low latency
•  No programming required
Flume:

“We just want to grab data
from this directory
and write it to HDFS”
We use Flume when:

•  Very high-throughput publish-subscribe messaging
•  Highly available
•  Stores data and can replay
•  Can support many consumers with no extra latency
Kafka is:

“Kafka is awesome.
We heard it cures cancer”
Use Kafka When:

•  Use Flume with a Kafka Source
•  Allows to get data from Kafka,
run some transformations
write to HDFS, HBase or Solr
Actually, why choose?

•  We want to ingest events from log files
•  Flume’s Spooling Directory source fits
•  With HDFS Sink
•  We would have used Kafka if…
–  We wanted the data in non-Hadoop systems too
In Our Example…

84
Sources Interceptors Selectors Channels Sinks
Flume Agent
Short Intro to Flume
Twitter, logs, JMS,
webserver, Kafka
Mask, re-format,
validate…
DR, critical
Memory, file,
Kafka
HDFS, HBase,
Solr

85
Configuration
•  Declarative
– No coding required.
– Configuration specifies
how components are
wired together.

86
Interceptors
•  Mask fields
•  Validate information
against external source
•  Extract fields
•  Modify data format
•  Filter or split events
©2014 Cloudera, Inc. All rights reserved.

Any sufficiently complex configuration
Is indistinguishable from code

88
A Brief Discussion of Flume Patterns – Fan-
in
•  Flume agent runs on
each of our servers.
•  These client agents
send data to multiple
agents to provide
reliability.
•  Flume provides support
for load balancing.

89
A Brief Discussion of Flume Patterns –
Splitting
•  Common need is to split
data on ingest.
•  For example:
– Sending data to multiple
clusters for DR.
– To multiple destinations.
•  Flume also supports
partitioning, which is key
to our implementation.

90
Flume Agent
Web Logs
Spooling
Dir
Source
Timestamp
Interceptor
File
Channel
Avro Sink
Avro Sink
Avro Sink
Flume Architecture – Client Tier
Web Server Flume Agent

91
Flume Architecture – Collector Tier
Flume Agent
Flume Agent
HDFS
Avro
Source
File
Channel
HDFS Sink
HDFS Sink

•  Add Kafka producer to our webapp
•  Send clicks and searches as messages
•  Flume can ingest events from Kafka
•  We can add a second consumer for real-time
processing in SparkStreaming
•  Another consumer for alerting…
•  And maybe a batch consumer too
What if…. We were to use Kafka?

93
Channels Sinks
Flume Agent
The Kafka Channel
Kafka
HDFS, HBase,
Solr
Producer
A
Producer
B
Producer
C
Kafka
Producers

94
Sources Interceptors Channels
Flume Agent
The Kafka Channel
Twitter, logs, JMS,
webserver
Mask, re-format,
validate…
Kafka
Consumer
A
Kafka
Consumers
Consumer
B
Consumer
C

95
Sources Interceptors Selectors Channels Sinks
Flume Agent
The Kafka Channel
Twitter, logs, JMS,
webserver
Mask, re-format,
validate…
DR, critical Kafka
HDFS, HBase,
Solr

96
Architectural
Considerations
Data Processing – Engines
tiny.cloudera.com/app-arch-slides

97
Processing Engines
•  MapReduce
•  Abstractions
•  Spark
•  Spark Streaming
•  Impala
Confidentiality Information Goes Here

98
MapReduce
•  Oldie but goody
•  Restrictive Framework / Innovated Work Around
•  Extreme Batch

99
MapReduce Basic High Level
Mapper
HDFS
(Replicated)
Native File System
Block of
Data
Temp Spill
Data
Partitioned
Sorted Data
Reducer
Reducer
Local Copy
Output File

100
MapReduce Innovation
•  Mapper Memory Joins
•  Reducer Memory Joins
•  Buckets Sorted Joins
•  Cross Task Communication
•  Windowing
•  And Much More

101
Abstractions
•  SQL
–  Hive
•  Script/Code
–  Pig: Pig Latin
–  Crunch: Java/Scala
–  Cascading: Java/Scala

102
Spark
•  The New Kid that isn’t that New Anymore
•  Easily 10x less code
•  Extremely Easy and Powerful API
•  Very good for machine learning
•  Scala, Java, and Python
•  RDDs
•  DAG Engine

103
Spark - DAG

104
Spark - DAG
Filter KeyBy
KeyBy
TextFile
TextFile
Join Filter Take

105
Spark - DAG
Filter KeyBy
KeyBy
TextFile
TextFile
Join Filter Take
Good
Good
Good
Good
Good
Good
Good-Replay
Good-Replay
Good-Replay
Good
Good-Replay
Good
Good-Replay
Lost Block
Replay
Good-Replay
Lost Block
Good
Future
Future
Future
Future

106
Spark Streaming
•  Calling Spark in a Loop
•  Extends RDDs with DStream
•  Very Little Code Changes from ETL to Streaming

107
Spark Streaming
Single Pass
Source Receiver RDD
Source Receiver RDD
RDD
Filter Count Print
Source Receiver RDD
RDD
RDD
Single Pass
Filter Count Print
Pre-first
Batch
First
Batch
Second
Batch

108
Spark Streaming
Single Pass
Source Receiver RDD
Source Receiver RDD
RDD
Filter Count
Print
Source Receiver RDD
RDD
RDD
Single Pass
Filter Count
Pre-first
Batch
First
Batch
Second
Batch
Stateful RDD 1
Print
Stateful RDD 2
Stateful RDD 1

109
Impala
•  MPP Style SQL Engine on top of Hadoop
•  Very Fast
•  High Concurrency
•  Analytical windowing functions (C5.2).

110
Impala – Broadcast Join
Impala Daemon
Smaller Table
Data Block
100% Cached
Smaller Table
Smaller Table
Data Block
Impala Daemon
100% Cached
Smaller Table
Impala Daemon
100% Cached
Smaller Table
Impala Daemon
Hash Join Function
Bigger Table
Data Block
100% Cached
Smaller Table
Output
Impala Daemon
Hash Join Function
Bigger Table
Data Block
100% Cached
Smaller Table
Output
Impala Daemon
Hash Join Function
Bigger Table
Data Block
100% Cached
Smaller Table
Output

111
Impala – Partitioned Hash Join
Impala Daemon
Smaller Table
Data Block
~33% Cached
Smaller Table
Smaller Table
Data Block
Impala Daemon
~33% Cached
Smaller Table
Impala Daemon
~33% Cached
Smaller Table
Hash Partitioner Hash Partitioner
Impala Daemon
BiggerTable
Data Block
Impala Daemon Impala Daemon
Hash Partitioner
Hash Join Function
33% Cached
Smaller Table
Hash Join Function
33% Cached
Smaller Table
Hash Join Function
33% Cached
Smaller Table
Output Output Output
BiggerTable
Data Block
Hash Partitioner
BiggerTable
Data Block
Hash Partitioner

112
Impala vs Hive
•  Very different approaches and
•  We may see convergence at some point
•  But for now
–  Impala for speed
–  Hive for batch

113
Architectural
Considerations
Data Processing – Patterns and
Recommendations

114
What processing needs to happen?
•  Sessionization
•  Filtering
•  Deduplication
•  BI / Discovery

115
Sessionization
Website visit
Visitor 1
Session 1
Visitor 1
Session 2
Visitor 2
Session 1
> 30 minutes

116
Why sessionize?
Helps answers questions like:
•  What is my website’s bounce rate?
–  i.e. how many % of visitors don’t go past the landing page?
•  Which marketing channels (e.g. organic search, display ad, etc.) are
leading to most sessions?
–  Which ones of those lead to most conversions (e.g. people buying things,
signing up, etc.)
•  Do attribution analysis – which channels are responsible for most
conversions?

117
Sessionization
244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://
bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36”
244.157.45.12+1413580110
244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?productID=1023 HTTP/
1.0" 200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U; Android 2.3.5;
en-us; HTC Vision Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0
Mobile Safari/533.1” 244.157.45.12+1413583199

118
How to Sessionize?
1.  Given a list of clicks, determine which clicks
came from the same user (Partitioning,
ordering)
2.  Given a particular user's clicks, determine if a
given click is a part of a new session or a
continuation of the previous session (Identifying
session boundaries)

119
#1 – Which clicks are from same user?
•  We can use:
–  IP address (244.157.45.12)
–  Cookies (A9A3BECE0563982D)
–  IP address (244.157.45.12)and user agent string ((KHTML, like
Gecko) Chrome/36.0.1944.0 Safari/537.36")

120
•  We can use:
–  IP address (244.157.45.12)
–  Cookies (A9A3BECE0563982D)
–  IP address (244.157.45.12)and user agent string ((KHTML, like
Gecko) Chrome/36.0.1944.0 Safari/537.36")

121
bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36”
244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?productID=1023 HTTP/1.0"
200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; HTC
Vision Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1”

122
#2 – Which clicks part of the same session?
> 30 mins apart = different
sessions

Sessionization engine recommendation
•  We have sessionization code in MR and Spark on github. The
complexity of the code varies, depends on the expertise in the
organization.
•  We choose MR
–  MR API is stable and widely known
–  No Spark + Oozie (orchestration engine) integration currently

124
Filtering – filter out incomplete records
200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U…

125
Filtering – filter out records from bots/spiders
Google spider IP address

Filtering recommendation
•  Bot/Spider filtering can be done easily in any of the engines
•  Incomplete records are harder to filter in schema systems like
Hive, Impala, Pig, etc.
•  Flume interceptors can also be used
•  Pretty close choice between MR, Hive and Spark
•  Can be done in Spark using rdd.filter()
•  We can simply embed this in our MR sessionization job

127
Deduplication – remove duplicate records

Deduplication recommendation
•  Can be done in all engines.
•  We already have a Hive table with all the columns, a simple
DISTINCT query will perform deduplication
•  reduce() in spark
•  We use Pig

BI/Discovery engine recommendation
•  Main requirements for this are:
–  Low latency
–  SQL interface (e.g. JDBC/ODBC)
–  Users don’t know how to code
•  We chose Impala
–  It’s a SQL engine
–  Much faster than other engines
–  Provides standard JDBC/ODBC interfaces

End-to-end processing
SessionizationFilteringDeduplication
BI tools

131
Architectural
Considerations
Orchestration

132
•  Data arrives through Flume
•  Triggers a processing event:
–  Sessionize
–  Enrich – Location, marketing channel…
–  Store as Parquet
•  Each day we process events from the previous day
Orchestrating Clickstream

•  Workflow is fairly simple
•  Need to trigger workflow based on data
•  Be able to recover from errors
•  Perhaps notify on the status
•  And collect metrics for reporting
Choosing Right

134
Oozie or Azkaban?

Oozie Architecture

•  Part of all major Hadoop distributions
•  Hue integration
•  Built -in actions – Hive, Sqoop, MapReduce, SSH
•  Complex workflows with decisions
•  Event and time based scheduling
•  Notifications
•  SLA Monitoring
•  REST API
Oozie features

•  Overhead in launching jobs
•  Steep learning curve
•  XML Workflows
Oozie Drawbacks

Azkaban Architecture
Azkaban Executor ServerAzkaban Web Server
HDFS viewer
plugin
Job types plugin
MySQL
Client
Hadoop

•  Simplicity
•  Great UI – including pluggable visualizers
•  Lots of plugins – Hive, Pig…
•  Reporting plugin
Azkaban features

•  Doesn’t support workflow decisions
•  Can’t represent data dependency
Azkaban Limitations

Choosing…
Easier in Oozie

Choosing the right Orchestration Tool
Better in Azkaban

The best orchestration tool
is the one you are an expert on
Important Decision Consideration!

Orchestration Patterns – Fan Out

Capture & Decide Pattern

146
Putting It All
Together
Final Architecture

147
Final Architecture – High Level Overview
Data
Sources
Ingestion
Raw Data
Storage
(Formats,
Schema)
Processed
Data Storage
(Formats,
Schema)
Processing
Data
Consumption
Orchestration
(Scheduling,
Managing,
Monitoring)

148
Data
Sources
Ingestion
Raw Data
Storage
(Formats,
Schema)
Processed
Data Storage
(Formats,
Schema)
Processing
Data
Consumption
Orchestration
(Scheduling,
Managing,
Monitoring)

149
Final Architecture – Ingestion/Storage
Flume Agent
Flume Agent
Flume Agent
Flume Agent
Fan-in
Pattern
Multi Agents for
Failover and rolling
restarts
HDFS
/etl/BI/casualcyclist/
clicks/rawlogs/
year=2014/month=10/
day=10!

150
Data
Sources
Ingestion
Raw Data
Storage
(Formats,
Schema)
Processed
Data Storage
(Formats,
Schema)
Processin
g
Data
Consumption
Orchestration
(Scheduling,
Managing,
Monitoring)

151
Final Architecture – Processing and Storage
/etl/BI/
casualcyclist/
clicks/rawlogs/
year=2014/
month=10/day=10
…
dedup->filtering-
>sessionization
/data/bikeshop/
clickstream/
year=2014/
month=10/day=10
…
parquetize

152
Data
Sources
Ingestion
Raw Data
Storage
(Formats,
Schema)
Processed
Data Storage
(Formats,
Schema)
Processing
Data
Consumption
Orchestration
(Scheduling,
Managing,
Monitoring)

153
Final Architecture – Data Access
Hive/
Impala
BI/
Analytics
Tools
DWH
Sqoop
Local
Disk
R, etc.
DB import tool
JDBC/ODBC

154
Demo

155Confidentiality Information Goes Here
Join the Discussion
Get community
help or provide
feedback
cloudera.com/community

156Try
Cloudera
Live
today—cloudera.com/live

157
BOOK SIGNINGS THEATER SESSIONS
TECHNICAL DEMOS GIVEAWAYS
Visit us at
Booth
#809
HIGHLIGHTS:
Apache Kafka is now fully
supported with Cloudera
Learn why Cloudera is the
leader for security in
Hadoop

158
Free books and Lots of Q&A!
•  Ask Us Anything!
–  Feb 19th, 1:30-2:10PM in 211 B
•  Book signings
–  Feb 19th, 11:15PM in Expo Hall – Cloudera Booth (#809)
–  Feb 19th, 3:00PM in Expo Hall - O'Reilly Booth
•  Stay in touch!
–  @hadooparchbook
–  hadooparchitecturebook.com
–  slideshare.com/hadooparchbook

Architectural considerations for Hadoop Applications

Recommended

More Related Content

What's hot (20)

Similar to Architectural considerations for Hadoop Applications (20)

More from hadooparchbook (6)

Recently uploaded (20)

Architectural considerations for Hadoop Applications