Why hadoop map reduce needs scala, an introduction to scoobi and scalding
7 likes2,984 views
This document provides an overview and comparison of Scoobi and Scalding, which are Scala libraries for implementing a higher-level programming model for Hadoop MapReduce. Scoobi uses source code generation to turn manipulations of distributed collection objects into MapReduce jobs, while Scalding wraps around Cascading. Both aim to provide a more intuitive and familiar Scala-based approach compared to lower-level MapReduce programming. The document discusses features of each library and notes that while they have similar capabilities, Scoobi may be closer to idiomatic Scala syntax while Scalding is more widely used due to Twitter's involvement in its development.
![package cascadingtutorial.wordcount;
/**
Very powerful!
* Wordcount example in Cascading
*/ Record Model
public class Main
{
Pipes & Filters
public static void main( String[] args )
{
String inputPath = args[0];
Joins & CoGroups
String outputPath = args[1];
Scheme inputScheme = new TextLine(new Fields("offset", "line"));
Scheme outputScheme = new TextLine();
Tap sourceTap = inputPath.matches( "^[^:]+://.*") ?
new Hfs(inputScheme, inputPath) : Not very intuitive
new Lfs(inputScheme, inputPath);
Tap sinkTap = outputPath.matches("^[^:]+://.*") ?
new Hfs(outputScheme, outputPath) : Strange new abstraction
new Lfs(outputScheme, outputPath);
Pipe wcPipe = new Each("wordcount",
Lots of boilerplate code
new Fields("line"),
new RegexSplitGenerator(new Fields("word"), "s+"),
new Fields("word"));
wcPipe = new GroupBy(wcPipe, new Fields("word"));
wcPipe = new Every(wcPipe, new Count(), new Fields("count", "word"));
Properties properties = new Properties();
FlowConnector.setApplicationJarClass(properties, Main.class);
Flow parsedLogFlow = new FlowConnector(properties)
.connect(sourceTap, sinkTap, wcPipe);
parsedLogFlow.start();
parsedLogFlow.complete();
}
}](https://image.slidesharecdn.com/whyhadoopmapreduceneedsscalaanintroductiontoscoobiandscalding-120611030137-phpapp01/85/Why-hadoop-map-reduce-needs-scala-an-introduction-to-scoobi-and-scalding-11-320.jpg)
![DList[T]
• Abstracts storage of data and files on HDFS
• Calling methods on DList objects to transform and
manipulate them abstracts the mapper, combiner,
sort-and-shuffle, and reducer phases of MapReduce
• Persisting a DList triggers compilation of the graph
into one or more MR jobs and their execution
• Very familiar: like standard Scala Lists
• Strongly typed
• Parameterized with rich types and Tuples
• Easy list manipulation using typical higher order
functions like map, flatMap, filter, etc.](https://image.slidesharecdn.com/whyhadoopmapreduceneedsscalaanintroductiontoscoobiandscalding-120611030137-phpapp01/85/Why-hadoop-map-reduce-needs-scala-an-introduction-to-scoobi-and-scalding-20-320.jpg)
![DList[T]](https://image.slidesharecdn.com/whyhadoopmapreduceneedsscalaanintroductiontoscoobiandscalding-120611030137-phpapp01/85/Why-hadoop-map-reduce-needs-scala-an-introduction-to-scoobi-and-scalding-21-320.jpg)
![IO
• Can read/write text files, Sequence files and Avro files
• Can influence sorting (raw, secondary)
Serialization
• Serialization of custom types through Scala type
classes and WireFormat[T]
• Scoobi implements WireFormat[T] for primitive types,
strings, tuples, Option[T], either[T], Iterable[T], etc.
• Out of the box support for serialization of Scala case
classes](https://image.slidesharecdn.com/whyhadoopmapreduceneedsscalaanintroductiontoscoobiandscalding-120611030137-phpapp01/85/Why-hadoop-map-reduce-needs-scala-an-introduction-to-scoobi-and-scalding-22-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Why hadoop map reduce needs scala, an introduction to scoobi and scalding (20)
Ad
More from Xebia Nederland BV (20)
Ad
Recently uploaded (20)
Why hadoop map reduce needs scala, an introduction to scoobi and scalding
- 1. Why Scalding Needs Scala A Look at Scoobi and Scalding Scala DSLs for Hadoop Scoobi @agemooij
- 2. Obligatory “About Me” Slide
- 3. Rocks!
- 4. But programming kinda Sucks!
- 5. Hello World Word Count using Hadoop MapReduce
- 6. Split lines into words Turn each word into a Pair(word, 1) Group by word (?) For each word, sum the 1s to get the total
- 7. Lots of small unintuitive Mapper and Reducer Classes Lots of Hadoop intrusiveness (Context, Writables, Exceptions, etc.) Low level glue code Actually runs the code on the cluster
- 8. This does not make me a happy Hadoop developer! Especially for things that are a little bit more complicated than counting words • Unintuitive, invasive programming model • Hard to compose/chain jobs into real, more complicated programs • Lots of low-level boilerplate code • Branching, Joins, CoGroups, etc. hard to implement
- 9. What Are the Alternatives?
- 10. Counting Words using Apache Pig Nice! Already a lot better, but anything more complex gets hard pretty fast. Pig is hard to customize/extend Handy for quick exploration of data! And the same goes for Hive
- 11. package cascadingtutorial.wordcount; /** Very powerful! * Wordcount example in Cascading */ Record Model public class Main { Pipes & Filters public static void main( String[] args ) { String inputPath = args[0]; Joins & CoGroups String outputPath = args[1]; Scheme inputScheme = new TextLine(new Fields("offset", "line")); Scheme outputScheme = new TextLine(); Tap sourceTap = inputPath.matches( "^[^:]+://.*") ? new Hfs(inputScheme, inputPath) : Not very intuitive new Lfs(inputScheme, inputPath); Tap sinkTap = outputPath.matches("^[^:]+://.*") ? new Hfs(outputScheme, outputPath) : Strange new abstraction new Lfs(outputScheme, outputPath); Pipe wcPipe = new Each("wordcount", Lots of boilerplate code new Fields("line"), new RegexSplitGenerator(new Fields("word"), "s+"), new Fields("word")); wcPipe = new GroupBy(wcPipe, new Fields("word")); wcPipe = new Every(wcPipe, new Count(), new Fields("count", "word")); Properties properties = new Properties(); FlowConnector.setApplicationJarClass(properties, Main.class); Flow parsedLogFlow = new FlowConnector(properties) .connect(sourceTap, sinkTap, wcPipe); parsedLogFlow.start(); parsedLogFlow.complete(); } }
- 12. Meh... I’m lazy I want more power with less work!
- 13. How would we count words in plain Scala? (My current language of choice)
- 15. But that code doesn’t scale to my cluster! Or does it? Meanwhile at Google...
- 16. Introducing Scoobi & Scalding Scala DSLs for Hadoop MapReduce NOTE: My relative familiarity with either platform: Scalding 5% Scoobi 95%
- 17. http://github.com/nicta/scoobi A Scala library that implements a higher level programming model for Hadoop MapReduce
- 18. Counting Words using Scoobi Split lines into words Turn each word into a Pair(word, 1) Group by word For each word, sum the 1s to get the total Actually runs the code on the cluster
- 19. Scoobi is... • A distributed collections abstraction: • Distributed collection objects abstract data in HDFS • Methods on these objects abstract map/reduce operations • Programs manipulate distributed collections objects • Scoobi turns these manipulations into MapReduce jobs • Based on Google’s FlumeJava / Cascades • A source code generator (it generates Java code!) • A job plan optimizer • Open sourced by NICTA • Written in Scala (W00t!)
- 20. DList[T] • Abstracts storage of data and files on HDFS • Calling methods on DList objects to transform and manipulate them abstracts the mapper, combiner, sort-and-shuffle, and reducer phases of MapReduce • Persisting a DList triggers compilation of the graph into one or more MR jobs and their execution • Very familiar: like standard Scala Lists • Strongly typed • Parameterized with rich types and Tuples • Easy list manipulation using typical higher order functions like map, flatMap, filter, etc.
- 21. DList[T]
- 22. IO • Can read/write text files, Sequence files and Avro files • Can influence sorting (raw, secondary) Serialization • Serialization of custom types through Scala type classes and WireFormat[T] • Scoobi implements WireFormat[T] for primitive types, strings, tuples, Option[T], either[T], Iterable[T], etc. • Out of the box support for serialization of Scala case classes
- 24. IO/Serialization II For normal (i.e. non-case) classes
- 25. Further Info Version 0.4 released today (!) • Avro, Sequence Files • Materialized DObjects • DList reduction methods (product, min, etc.) • Vastly improved testing support • Less overhead • Much more http://nicta.github.com/scoobi/ [email protected] [email protected]
- 26. Scalding! http://github.com/twitter/scalding A Scala library that implements a higher level programming model for Hadoop MapReduce Cascading
- 27. Counting Words using Scalding
- 28. Scalding is... • A distributed collections abstraction • A wrapper around Cascading (i.e. no source code generation) • Based on the same record model (i.e. named fields) • Less strongly typed • Uses Kryo Serialization • Used by Twitter in production • Written in Scala (W00t!)
- 29. Further Info Current version: 0.5.4 http://github.com/twitter/scalding https://github.com/twitter/scalding/wiki @scalding [email protected] http://blog.echen.me/2012/02/09/movie-recommendations-and-more- via-mapreduce-and-scalding/
- 30. How do they compare? Small feature Different approaches, differences, which will similar power even out over time Scoobi gets a little Twitter is definitely a closer to idiomatic bigger fish than Scala NICTA, so Scalding gets all the attention Both open sourced (last year) Scoobi has better docs!
- 31. Which one should I use? Ehm... ...I’m extremely prejudiced!
- 32. Questions?