Scribd
Upload
206 views

Building 1000 Node Spark Cluster On EMR

The document discusses building a 1000 node cluster on Amazon EMR (Elastic MapReduce). It provides an overview of EMR concepts including master nodes, core nodes, and task nodes. It also covers using Spark on EMR including installing Spark via bootstrap actions, scaling the cluster by adding task nodes for more memory and removing them after jobs complete, and autoscaling based on memory usage. It provides an example command to easily launch a 1000 node Spark/Shark cluster on EMR.

Uploaded by

Anonymous iNxLvw
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
206 views

Building 1000 Node Spark Cluster On EMR

The document discusses building a 1000 node cluster on Amazon EMR (Elastic MapReduce). It provides an overview of EMR concepts including master nodes, core nodes, and task nodes. It also covers using Spark on EMR including installing Spark via bootstrap actions, scaling the cluster by adding task nodes for more memory and removing them after jobs complete, and autoscaling based on memory usage. It provides an example command to easily launch a 1000 node Spark/Shark cluster on EMR.

Uploaded by

Anonymous iNxLvw
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 53

Building

1000 node cluster on


EMR
Manjeet Chayel

What is EMR?
Amazon Elas+c MapReduce

Hadoop-as-a-service

Map-Reduce engine

Massively parallel


What is EMR?

Integrated with tools

Integrated to AWS services


Cost eecBve AWS wrapper

HDFS

Amazon EMR

HDFS

Amazon EMR

AWS Data Pipeline


Amazon S3

Amazon
DynamoDB

HDFS

Data
Sources

Amazon EMR
Amazon
Kinesis

AWS Data Pipeline


Amazon S3

Amazon
DynamoDB

Data management

Hadoop Ecosystem analyBcal tools

HDFS

Data
Sources

Amazon EMR
Amazon
Kinesis

AWS Data Pipeline


Amazon S3

Amazon
DynamoDB

Data management

Hadoop Ecosystem analyBcal tools

HDFS

Data
Sources

Amazon
RDS

Amazon EMR
Amazon
Kinesis

AWS Data Pipeline


Amazon
RedShift

Amazon S3

Amazon
DynamoDB

Data management

Hadoop Ecosystem analyBcal tools

HDFS

Data
Sources

Amazon
RDS

Amazon EMR
Amazon
Kinesis

AWS Data Pipeline


Amazon
RedShift

Amazon S3

Amazon
DynamoDB

AWS Data
Pipeline

Amazon EMR Concepts


Master Node
Core Nodes
Task Nodes

Core Nodes
Amazon EMR cluster




DataNode (HDFS)

Master instance group


Master
Node

HDFS

HDFS

Core instance group

Core Nodes
Amazon EMR cluster




Can Add Core Nodes:

More CPU

More Memory

More HDFS Space

Master instance group


Master
Node

HDFS

HDFS

Core instance group

HDFS

Core Nodes
Amazon EMR cluster




Cant remove core
nodes:

HDFS corrupBon

Master instance group


Master
Node

HDFS

HDFS

Core instance group

HDFS

Task Nodes
Amazon EMR cluster




No HDFS

Provides compute
resources:

CPU

Memory

Master instance group


Master
Node

HDFS

HDFS

Core instance group

Task Nodes
Amazon EMR cluster




Can add and remove
task nodes

Master instance group


Master
Node

HDFS

HDFS

Core instance group

Spark On Amazon EMR

Bootstrap AcBons
Ability to run or install addiBonal packages/
soUware on EMR nodes
Simple bash script stored on S3
Script gets executed during node/instance boot
Bme
Script gets executed on every node that gets
added to the cluster

Spark on Amazon EMR


Bootstrap acBon installs Spark on EMR nodes
Currently on Spark 0.8.1 & upgrading to 1.0
very soon

Why Spark on Amazon EMR?


Deploy small and large Spark clusters in
minutes
EMR manages your cluster, and handles node
recover in case of failures
IntegraBon with EC2 Spot Market, Amazon
RedshiU, Amazon Data pipeline, Amazon
Cloudwatch and etc
Tight integraBon with Amazon S3

Why Spark on Amazon EMR?


Shipping Spark logs to S3 for debugging
Dene S3 bucket at cluster deploy Bme

Scaling Spark on EMR


Amazon EMR cluster
Master instance group
Master
Node

Launch iniBal
Spark cluster with
core nodes
HDFS to store and
checkpoint RDDs

HDFS

HDFS

32GB Memory

Scaling Spark on EMR


Amazon EMR cluster
Master instance group
Master
Node

Add Task nodes in


spot market to
increase memory
capacity
HDFS

HDFS

32GB Memory

256GB Memory

Scaling Spark on EMR


Create RDDs from
HDFS or Amazon S3
with:

sc.textFile
OR
sc.sequenceFile


Run ComputaBon on
RDDs

Amazon EMR cluster


Master instance group
Master
Node

HDFS

HDFS

32GB Memory

256GB Memory

Amazon S3

Scaling Spark on EMR


Amazon EMR cluster
Master instance group

Save the resulBng


RDDs to HDFS or S3
with:

rdd.saveAsSequenceFile
OR
rdd.saveAsObjectFile

Master
Node

HDFS

HDFS

32GB Memory

256GB Memory
saveAsObjectFile

Amazon S3

Scaling Spark on EMR


Amazon EMR cluster
Master instance group
Master
Node

Shutdown
TaskNodes when
your job is done
HDFS

HDFS

32GB Memory

ElasBc Spark With Amazon EMR

Autoscaling Spark
Amazon EMR cluster
Master Node

HDFS

HDFS

32GB Memory

Autoscaling Spark
Amazon EMR cluster
Master Node

HDFS

HDFS

32GB Memory

256GB Memory

ElasBc Spark
When to Scale?
Depends on your job

CPU bounded or Memory intensive?


Probably both for Spark jobs

Use CPU/Memory uBl. metrics to decide when to


scale

Spark Autoscaling Based Memory


Spark needs memory
Lots of it!!

How to scale based on the memory usage?


Launching a 1000 node Cluster



Amazon EMR

Launching a 1000 node Cluster


What do I need to launch a cluster?
AWS Account
Amazon EMR CLI

Launching a 1000 node Cluster


Easy to launch 1 command
./elas2c-mapreduce --create alive
--name "Spark/Shark Cluster" \
--bootstrap-ac2on s3://elasBcmapreduce/samples/spark/0.8.1/install-spark-shark.sh
--bootstrap-name "Spark/Shark"
--instance-type m1.xlarge
--instance-count 1000

Comes up in 15-20mins

Launching a 1000 node Cluster


Adding Task nodes

--add-instance-group TASK
--instance-count INSTANCE_COUNT
--instance-type INSTANCE_TYPE

Is cluster ready?

Cluster will be in WaiBng state

Lynx Interface
lynx hhp://localhost:9101

Web Interface

Spark UI

Dataset
Wikipedia arBcle trac staBsBcs
4.5 TB
104 Billion records


Stored in Amazon S3
s3://bigdata-spark-demo/wikistats/

Dataset
File structure (pagecount-DATE-HOUR.gz)
Period: Dec-2007 to Feb-2014
Format of File (tsv)
Feilds
projectcode, pagename, pageviews, and bytes

Sample dataset
Projectcode pagename pageviews bytes






en Barack_Obama 997 123091092


en Barack_Obama%27s_rst_100_days 8 850127
en Barack_Obama,_Jr 1 144103
en Barack_Obama,_Sr. 37 938821
en Barack_Obama_%22HOPE%22_poster 4 81005
en Barack_Obama_%22Hope%22_poster 5 102081

Loading data
HDFS

Amazon EMR

Amazon S3

Analyze opBons

Table structure
create external table wikistats
(
projectcode string,
pagename string,
pageviews int,
pagesize int
)
ROW FORMAT
DELIMITED FIELDS
TERMINATED BY ' '
LOCATION 's3n://bigdata-spark-demo/wikistats/';

ALTER TABLE wikistats add parBBon(dt=2007-12) locaBon 's3n://bigdata-spark-
demo//wikistats/2007/2007-12';
.
Adding parBBons for every month Bll 2014-04

Analyze using Shark


Top 10 Page Views in Jan 2014




Exec Bme: 26 secs
Scanning 250GB of data

Analyze using Shark


Query 2:
Top 10 Page Views Overall




Exec Bme: 45 sec
Scanning 4.5TB of data

Analyze using Shark


Query 3:
No of pages in each projectcodes




Exec Bme: 48 secs
Scanning 4.5TB of data / 104 Billion records

Spark Streaming and Amazon


Kinesis

Amazon Kinesis
CreateStream
Creates a new Data Stream within the Kinesis Service
PutRecord
Adds new records to a Kinesis Stream
DescribeStream
Provides metadata about the Stream, including name, status, Shards,
etc.
GetNextRecord
Fetches next record for processing by user business logic
MergeShard / SplitShard
Scales Stream up/ down
DeleteStream
Deletes the Stream

Amazon Kinesis
Kinesis

Amazon Kinesis
Kinesis

What Do You Like To See?



Send Feedbacks To:

Manjeet Chayel
[email protected]
hhp://bit.ly/sparkemr

You might also like