20140614 introduction to spark-ben white
3 likes1,062 views
This document provides an introduction to Apache Spark. It begins by explaining how Spark improves upon MapReduce by leveraging distributed memory for better performance and supporting iterative algorithms. Spark is described as a general purpose computational framework that retains the advantages of MapReduce like scalability and fault tolerance, while offering more functionality through directed acyclic graphs and libraries for machine learning. The document then discusses getting started with Spark and its execution modes like standalone, YARN client, and YARN cluster. Finally, it introduces Spark concepts like Resilient Distributed Datasets (RDDs), which are collections of objects partitioned across a cluster.
![Parallelized
Collec6ons
14
scala>
val
data
=
1
to
5
data:
Range.Inclusive
=
Range(1,
2,
3,
4,
5)
scala>
val
distData
=
sc.parallelize(data)
distData:
org.apache.spark.rdd.RDD[Int]
=
ParallelCollectionRDD[0]
Now
I
can
apply
parallel
opera6ons
to
this
array:
scala>
distData.reduce(_
+
_)
[…
Adding
task
set
0.0
with
56
tasks
…]
res0:
Int
=
15
What
just
happened?!](https://image.slidesharecdn.com/20140614-introductiontospark-benwhite-140623121120-phpapp01/85/20140614-introduction-to-spark-ben-white-14-320.jpg)
![Fault
Tolerance
• RDDs
contain
lineage.
• Lineage
–
source
loca6on
and
list
of
transforma6ons
• Lost
par66ons
can
be
re-‐computed
from
source
data
18
msgs = textFile.filter(lambda s: s.startsWith(“ERROR”))
.map(lambda s: s.split(“t”)[2])
HDFS
File
Filtered
RDD
Mapped
RDD
filter
(func
=
startsWith(…))
map
(func
=
split(...))](https://image.slidesharecdn.com/20140614-introductiontospark-benwhite-140623121120-phpapp01/85/20140614-introduction-to-spark-ben-white-18-320.jpg)
![Word
Count
in
MapReduce
20
package
org.myorg;
import
java.io.IOExcep6on;
import
java.u6l.*;
import
org.apache.hadoop.fs.Path;
import
org.apache.hadoop.conf.*;
import
org.apache.hadoop.io.*;
import
org.apache.hadoop.mapreduce.*;
import
org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import
org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public
class
WordCount
{
public
sta6c
class
Map
extends
Mapper<LongWritable,
Text,
Text,
IntWritable>
{
private
final
sta6c
IntWritable
one
=
new
IntWritable(1);
private
Text
word
=
new
Text();
public
void
map(LongWritable
key,
Text
value,
Context
context)
throws
IOExcep6on,
InterruptedExcep6on
{
String
line
=
value.toString();
StringTokenizer
tokenizer
=
new
StringTokenizer(line);
while
(tokenizer.hasMoreTokens())
{
word.set(tokenizer.nextToken());
context.write(word,
one);
}
}
}
public
sta6c
class
Reduce
extends
Reducer<Text,
IntWritable,
Text,
IntWritable>
{
public
void
reduce(Text
key,
Iterable<IntWritable>
values,
Context
context)
throws
IOExcep6on,
InterruptedExcep6on
{
int
sum
=
0;
for
(IntWritable
val
:
values)
{
sum
+=
val.get();
}
context.write(key,
new
IntWritable(sum));
}
}
public
sta6c
void
main(String[]
args)
throws
Excep6on
{
Configura6on
conf
=
new
Configura6on();
Job
job
=
new
Job(conf,
"wordcount");
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job,
new
Path(args[0]));
FileOutputFormat.setOutputPath(job,
new
Path(args[1]));
job.waitForComple6on(true);
}
}](https://image.slidesharecdn.com/20140614-introductiontospark-benwhite-140623121120-phpapp01/85/20140614-introduction-to-spark-ben-white-20-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to 20140614 introduction to spark-ben white (20)
Ad
More from Data Con LA (20)
Recently uploaded (20)
20140614 introduction to spark-ben white
- 1. CONFIDENTIAL -‐ RESTRICTED Introduc6on to Spark Ben White – Systems Engineer, Cloudera
- 2. 2 But first… how did we get here?
- 3. What does Hadoop look like? 3 HDFS worker (“DN”) MR worker (“TT”) HDFS worker (“DN”) MR worker (“TT”) HDFS worker (“DN”) MR worker (“TT”) HDFS worker (“DN”) MR worker (“TT”) HDFS worker (“DN”) MR worker (“TT”) … HDFS master (“NN”) MR master (“JT”) Standby master
- 4. But I want MORE! 4 HDFS worker HDFS worker HDFS worker HDFS worker HDFS worker … MapReduce HDFS master (“NN”) MR master (“JT”) Standby master
- 5. Hadoop as an Architecture The Old Way $30,000+ per TB Expensive & UnaNainable • Hard to scale • Network is a boNleneck • Only handles rela6onal data • Difficult to add new fields & data types Expensive, Special purpose, “Reliable” Servers Expensive Licensed SoRware Network Data Storage (SAN, NAS) Compute (RDBMS, EDW) The Hadoop Way $300-‐$1,000 per TB Affordable & ANainable • Scales out forever • No boNlenecks • Easy to ingest any data • Agile data access Commodity “Unreliable” Servers Hybrid Open Source SoRware Compute (CPU) Memory Storage (Disk) z z
- 6. CDH: the App Store for Hadoop 6 Integra6on Storage Resource Management Metadata NoSQL DBMS … AnalyYc MPP DBMS Search Engine In-‐ Memory Batch Processing System Management Data Management Support Security Machine Learning MapReduce
- 7. 7 Introduc6on to Apache Spark Credits: • Todd Lipcon • Ted Malaska • Jairam Ranganathan • Jayant Shekhar • Sandy Ryza
- 8. Can we improve on MR? • Problems with MR: • Very low-‐level: requires a lot of code to do simple things • Very constrained: everything must be described as “map” and “reduce”. Powerful but some6mes difficult to think in these terms. 8
- 9. Can we improve on MR? • Two approaches to improve on MapReduce: 1. Special purpose systems to solve one problem domain well. • Giraph / Graphlab (graph processing) • Storm (stream processing) 2. Generalize the capabili6es of MapReduce to provide a richer founda6on to solve problems. • Tez, MPI, Hama/Pregel (BSP), Dryad (arbitrary DAGs) Both are viable strategies depending on the problem! 9
- 10. What is Apache Spark? Spark is a general purpose computa6onal framework Retains the advantages of MapReduce: • Linear scalability • Fault-‐tolerance • Data Locality based computa6ons …but offers so much more: • Leverages distributed memory for beNer performance • Supports itera6ve algorithms that are not feasible in MR • Improved developer experience • Full Directed Graph expressions for data parallel computa6ons • Comes with libraries for machine learning, graph analysis, etc 10
- 11. Gecng started with Spark • Java API • Interac6ve shells: • Scala (spark-‐shell) • Python (pyspark) 11
- 12. Execu6on modes • Standalone Mode • Dedicated master and worker daemons • YARN Client Mode • Launches a YARN applica6on with the driver program running locally • YARN Cluster Mode • Launches a YARN applica6on with the driver program running in the YARN Applica6onMaster 12 Dynamic resource management between Spark, MR, Impala… Dedicated Spark run6me with sta6c resource limits
- 14. Parallelized Collec6ons 14 scala> val data = 1 to 5 data: Range.Inclusive = Range(1, 2, 3, 4, 5) scala> val distData = sc.parallelize(data) distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] Now I can apply parallel opera6ons to this array: scala> distData.reduce(_ + _) [… Adding task set 0.0 with 56 tasks …] res0: Int = 15 What just happened?!
- 15. RDD – Resilient Distributed Dataset • Collec6ons of objects par66oned across a cluster • Stored in RAM or on Disk • You can control persistence and par66oning • Created by: • Distribu6ng local collec6on objects • Transforma6on of data in storage • Transforma6on of RDDs • Automa6cally rebuilt on failure (resilient) • Contains lineage to compute from storage • Lazy materializa6on 15
- 17. Opera6ons on RDDs TransformaYons lazily transform a RDD to a new RDD • map • flatMap • filter • sample • join • sort • reduceByKey • … AcYons run computa6on to return a value • collect • reduce(func) • foreach(func) • count • first, take(n) • saveAs • … 17
- 18. Fault Tolerance • RDDs contain lineage. • Lineage – source loca6on and list of transforma6ons • Lost par66ons can be re-‐computed from source data 18 msgs = textFile.filter(lambda s: s.startsWith(“ERROR”)) .map(lambda s: s.split(“t”)[2]) HDFS File Filtered RDD Mapped RDD filter (func = startsWith(…)) map (func = split(...))
- 19. 19 Examples
- 20. Word Count in MapReduce 20 package org.myorg; import java.io.IOExcep6on; import java.u6l.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; public class WordCount { public sta6c class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final sta6c IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws IOExcep6on, InterruptedExcep6on { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } public sta6c class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOExcep6on, InterruptedExcep6on { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } public sta6c void main(String[] args) throws Excep6on { Configura6on conf = new Configura6on(); Job job = new Job(conf, "wordcount"); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForComple6on(true); } }
- 21. Word Count in Spark ! sc.textFile(“words”)! .flatMap(line => line.split(" "))! .map(word=>(word,1))! .reduceByKey(_+_).collect()! ! ! 21
- 22. Logis6c Regression • Read two sets of points • Looks for a plane W that separates them • Perform gradient descent: • Start with random W • On each itera6on, sum a func6on of W over the data • Move W in a direc6on that improves it 22
- 23. Intui6on 23
- 26. 26 Spark and Hadoop: a Framework within a Framework
- 27. 27 Integra6on Storage Resource Management Metadata HBase … Impala Solr Spark Map Reduce System Management Data Management Support Security
- 28. 28
- 29. 29
- 30. 30 Integra6on Storage Resource Management Metadata HBase … Impala Solr Spark Map Reduce System Management Data Management Support Security
- 31. Spark Streaming • Takes the concept of RDDs and extends it to DStreams • Fault-‐tolerant like RDDs • Transformable like RDDs • Adds new “rolling window” opera6ons • Rolling averages, etc • But keeps everything else! • Regular Spark code works in Spark Streaming • Can s6ll access HDFS data, etc 31
- 32. Micro-‐batching for on the fly ETL 32
- 33. Fault recovery How fast can the system recover? 33
- 34. Fault Recovery • RDDs store dependency graph • Because RDDs are determinis6c: Missing RDDs are rebuilt in parallel on other nodes • Stateful RDDs can have infinite lineage • Periodic checkpoints to disk clears lineage • Faster recovery 6mes • BeNer handling of stragglers vs row-‐by-‐row streaming 34
- 35. 35 Summary
- 36. Why Spark? • Flexible like MapReduce • High performance • Machine learning, itera6ve algorithms • Interac6ve data explora6ons • Concise, easy API for developer produc6vity 36
- 37. 37
- 38. Spark 38 hNp://www.cloudera.com/content/cloudera/en/products-‐and-‐services/cdh/spark.html hNp://www.cloudera.com/content/cloudera-‐content/cloudera-‐docs/CM5/latest/Cloudera-‐ Manager-‐Installa6on-‐Guide/cm5ig_install_spark.html
- 39. A Brief History 39 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Doug Cu`ng launches Nutch project Google releases GFS paper Google releases MapReduce paper MapReduce implemented in Nutch Nutch adds distributed file system Hadoop spun out of Nutch project Hadoop breaks Terasort world record Cloudera founded CDH and CDH2 released CDH3 released CDH4 released adding HA Impala (SQL on Hadoop) launched Sentry and Search launched CDH5 Cloudera Manager released HBase, Zookeeper, Flume and more added to CDH
- 40. What is Apache Hadoop? • An open-‐source implementa6on of Google’s GFS and MapReduce papers • An Apache So~ware Founda6on top-‐level project • Good at storing and processing all kinds of data • Reliable storage at terabyte/petabyte-‐scale on unreliable (cheap) hardware • A distributed system for coun6ng words J 40
- 41. What is Apache Hadoop? 41 Has the Flexibility to Store and Mine Any Type of Data § Ask ques6ons across structured and unstructured data that were previously impossible to ask or solve § Not bound by a single schema Excels at Processing Complex Data § Scale-‐out architecture divides workloads across mul6ple nodes § Flexible file system eliminates ETL boNlenecks Scales Economically § Can be deployed on industry standard hardware § Open source pla•orm guards against vendor lock Hadoop Distributed File System (HDFS) Self-‐Healing, High Bandwidth Clustered Storage MapReduce Distributed Compu6ng Framework Apache Hadoop is an open source pla•orm for data storage and processing that is… ü Scalable ü Fault tolerant ü Distributed CORE HADOOP SYSTEM COMPONENTS