Barcelona MUG MongoDB + Hadoop Presentation
5 likes3,332 views
- The document discusses MongoDB and Hadoop, two popular big data platforms, and the MongoDB + Hadoop Connector which allows interoperation between the two. - It provides an overview of MongoDB and Hadoop's key features for scalability, availability and processing large datasets. - The connector allows processing data across MongoDB and Hadoop through MapReduce jobs without needing custom exports/imports. - Examples show building a graph of email sender/recipient relationships from an Enron dataset stored in MongoDB using Hadoop Streaming, Pig and Hive.
![@Override
public void map(NullWritable key, BSONObject val, final Context context){
BSONObject headers = (BSONObject)val.get("headers");
if(headers.containsKey("From") && headers.containsKey("To")){
String from = (String)headers.get("From"); String to = (String) headers.get("To");
String[] recips = to.split(",");
for(int i=0;i<recips.length;i++){
String recip = recips[i].trim();
context.write(new MailPair(from, recip), new IntWritable(1));
}
}
}
Map Phase – each document get’s
through mapper function](https://image.slidesharecdn.com/130930bcnmongodbhadoop-131001032649-phpapp02/85/Barcelona-MUG-MongoDB-Hadoop-Presentation-30-320.jpg)
![from pymongo_hadoop import BSONMapper
def mapper(documents): i=0
for doc in documents: i=i+1
from_field = doc['headers']['From']
to_field = doc['headers']['To']
recips = [x.strip() for x in to_field.split(',')]
for r in recips:
yield {'_id': {'f':from_field, 't':r}, 'count': 1}
BSONMapper(mapper)
print >> sys.stderr, "Done Mapping."
Map Phase](https://image.slidesharecdn.com/130930bcnmongodbhadoop-131001032649-phpapp02/85/Barcelona-MUG-MongoDB-Hadoop-Presentation-39-320.jpg)
![from pymongo_hadoop import BSONReducer
def reducer(key, values):
print >> sys.stderr, "Processing from/to %s" % str(key)
_count = 0
for v in values:
_count += v['count']
return {'_id': key, 'count': _count}
BSONReducer(reducer)
Reduce Phase](https://image.slidesharecdn.com/130930bcnmongodbhadoop-131001032649-phpapp02/85/Barcelona-MUG-MongoDB-Hadoop-Presentation-40-320.jpg)
![raw = LOAD 'hdfs:///messages.bson'
using com.mongodb.hadoop.pig.BSONLoader('','headers:[]') ;
send_recip = FOREACH raw GENERATE $0#'From' as from, $0#'To'
as to;
//filter && split
send_recip_filtered = FILTER send_recip BY to IS NOT NULL;
send_recip_split = FOREACH send_recip_filtered GENERATE
from as from, TRIM(FLATTEN(TOKENIZE(to))) as to;
//group && count
send_recip_grouped = GROUP send_recip_split BY (from, to);
send_recip_counted = FOREACH send_recip_grouped GENERATE
group, COUNT($1) as count;
STORE send_recip_counted INTO 'file:///enron_results.bson' using
com.mongodb.hadoop.pig.BSONStorage;
PIG](https://image.slidesharecdn.com/130930bcnmongodbhadoop-131001032649-phpapp02/85/Barcelona-MUG-MongoDB-Hadoop-Presentation-45-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Barcelona MUG MongoDB + Hadoop Presentation (20)
Ad
More from Norberto Leite (20)
Ad
Recently uploaded (20)
Barcelona MUG MongoDB + Hadoop Presentation
- 1. MongoDB + Hadoop Wingman, MongoDB Inc. Norberto Leite #bcnmug
- 2. Agenda • MongoDB • Hadoop • MongoDB + Hadoop Connector • How it works • What can we do with it
- 3. MongoDB
- 4. MongoDB The leading NoSQL database Document Database Open- Source General Purpose
- 5. Better Data Locality Performance In-Memory Caching In-Place Updates
- 6. Scalability Auto-Sharding • Increase capacity as you go • Commodity and cloud architectures • Improved operational simplicity and cost visibility
- 7. High Availability • Automated replication and failover • Multi-data center support • Improved operational simplicity (e.g., HW swaps) • Data durability and consistency
- 8. Shell Command-line shell for interacting directly with database Shell and Drivers Drivers Drivers for most popular programming languages and frameworks > db.collection.insert({product:“MongoDB”, type:“Document Database”}) > > db.collection.findOne() { “_id” : ObjectId(“5106c1c2fc629bfe52792e86”), “product” : “MongoDB” “type” : “Document Database” } Java Python Perl Ruby Haskell JavaScript
- 9. Hadoop
- 10. Hadoop • Google publishes seminal papers – GFS – global file system (Oct 2003) – MapReduce – divide and conquer (Dec 2004) – How they indexed the internet • Yahoo builds and open sources (2006) – Doug Cutting lead, now at Cloudera – Most others now at Hortonworks • Commonly mentioned has: – The elephant in the room!
- 11. Hadoop • Primary components – HDFS–Hadoop Distributed File System – MapReduce–parallel processing engine • Ecosystem – HIVE – HBASE – PIG – Oozie – Sqoop – Zookeeper
- 15. MongoDB Hadoop Connector • http://api.mongodb.org/hadoop/MongoDB %2BHadoop+Connector.html • Allows interoperation of MongoDB and Hadoop • “Give power to the people” • Allows processing across multiple sources • Avoid custom hacky exports and imports • Scalability and Flexibility to accommodate Hadoop and|or MongoDB changes
- 17. MongoDB with Hadoop• MongoDB warehouse
- 19. Benefits and Features • Full multi-core parallelism to process data in MongoDB • Full integration w/ Hadoop and JVM ecosystem • Can be used on Amazon Elastic MapReduce • Read and write backup files to local,HDFS and S3 • Vanilla Java MapReduce • But not only using Hadoop Streaming
- 20. Benefits and Features • Supports HIVE• Supports PIG
- 21. How it works • Adapter examines MongoDB input collection and calculates a set of splits from data • Each split is assigned to a Hadoop node • In parallel hadoop pulls data from splits on MongoDB (or BSON) and starts processing locally • Hadoop merges results and streams output back to MongoDB (or BSON) output collection
- 22. Example Tour
- 23. Tour bus stops • Java MapReduce w/ MongoDB-Hadoop Connector • Using Hadoop Streaming • Pig and Hive
- 24. Data Set • ENRON emails corpus (501 records,1.75GB) • Each document is one email • https://www.cs.cmu.edu/~enron/
- 25. { "_id" : ObjectId("4f2ad4c4d1e2d3f15a000000"), "body" : "Here is our forecastnn ", "filename" : "1.", "headers" : { "From" : "[email protected]", "Subject" : "Forecast Info", "X-bcc" : "", "To" : "[email protected]", "X-Origin" : "Allen-P", "X-From" : "Phillip K Allen", "Date" : "Mon, 14 May 2001 16:39:00 -0700 (PDT)", "X-To" : "Tim Belden ", "Message-ID" : "<18782981.1075855378110.JavaMail.evans@thyme>", "Content-Type" : "text/plain; charset=us-ascii", "Mime-Version" : "1.0" } } Document Example
- 26. Let’s build a graph between senders and recipients and count the messages exchanged
- 27. Graph Sketch
- 28. {"_id": {"t":"[email protected]", "f":"[email protected]"}, "count" : 14} {"_id": {"t":"[email protected]", "f":"[email protected]"}, "count" : 9} {"_id": {"t":"[email protected]", "f":"[email protected]"}, "count" : 99} {"_id": {"t":"[email protected]", "f":"[email protected]"}, "count" : 48} {"_id": {"t":"[email protected]", "f":"[email protected]"}, "count" : 20} Receiver Sender Pairs
- 30. @Override public void map(NullWritable key, BSONObject val, final Context context){ BSONObject headers = (BSONObject)val.get("headers"); if(headers.containsKey("From") && headers.containsKey("To")){ String from = (String)headers.get("From"); String to = (String) headers.get("To"); String[] recips = to.split(","); for(int i=0;i<recips.length;i++){ String recip = recips[i].trim(); context.write(new MailPair(from, recip), new IntWritable(1)); } } } Map Phase – each document get’s through mapper function
- 31. public void reduce( final MailPair pKey, final Iterable<IntWritable> pValues, final Context pContext ){ int sum = 0; for ( final IntWritable value : pValues ){ sum += value.get(); } BSONObject outDoc = new BasicDBObjectBuilder().start() .add( "f" , pKey.from) .add( "t" , pKey.to ) .get(); BSONWritable pkeyOut = new BSONWritable(outDoc); pContext.write( pkeyOut, new IntWritable(sum) ); } Reduce Phase – output Maps are grouped by key and passed to Reducer
- 32. mongo.job.input.format=com.mongodb.hadoop.MongoInputFormat mongo.input.uri=mongodb://my-db:27017/enron.messages mongo.job.input.format=com.mongodb.hadoop.BSONFileInputFormat mapred.input.dir= file:///tmp/messages.bson mapred.input.dir= hdfs:///tmp/messages.bson mapred.input.dir= s3:///tmp/messages.bson (or BSON)Read From MongoDB
- 33. mongo.job.output.format=com.mongodb.hadoop.MongoOutputFormat mongo.output.uri=mongodb://my-db:27017/enron.results_out mongo.job.output.format=com.mongodb.hadoop.BSONFileOutputFormat mapred.output.dir= file:///tmp/results.bson mapred.output.dir= hdfs:///tmp/results.bson mapred.output.dir= s3:///tmp/results.bson (or BSON)Write To MongoDB
- 34. mongos> db.streaming.output.find({"_id.t": /^kenneth.lay/}) { "_id" : { "t" : "[email protected]", "f" : "[email protected]" }, "count" : 1 } { "_id" : { "t" : "[email protected]", "f" : "[email protected]" }, "count" : 1 } { "_id" : { "t" : "[email protected]", "f" : "[email protected]" }, "count" : 2 } { "_id" : { "t" : "[email protected]", "f" : "[email protected]" }, "count" : 2 } { "_id" : { "t" : "[email protected]", "f" : "[email protected]" }, "count" : 4 } { "_id" : { "t" : "[email protected]", "f" : "[email protected]" }, "count" : 1 } { "_id" : { "t" : "[email protected]", "f" : "[email protected]" }, "count" : 1 } Query Data
- 35. Hadoop Streaming
- 36. Same thing but lets use Python PYTHON FTW!!!
- 37. Hadoop Streaming: How it works
- 38. > pip install pymongo_hadoop Install python library
- 39. from pymongo_hadoop import BSONMapper def mapper(documents): i=0 for doc in documents: i=i+1 from_field = doc['headers']['From'] to_field = doc['headers']['To'] recips = [x.strip() for x in to_field.split(',')] for r in recips: yield {'_id': {'f':from_field, 't':r}, 'count': 1} BSONMapper(mapper) print >> sys.stderr, "Done Mapping." Map Phase
- 40. from pymongo_hadoop import BSONReducer def reducer(key, values): print >> sys.stderr, "Processing from/to %s" % str(key) _count = 0 for v in values: _count += v['count'] return {'_id': key, 'count': _count} BSONReducer(reducer) Reduce Phase
- 41. Surviving Hadoop
- 42. MapReduce easier w/ PIG and Hive • PIG – Powerful language – Generates sophisticated map/reduce – Workflows from simple scripts • HIVE – Similar to PIG – SQL as language
- 43. data = LOAD 'mongodb://localhost:27017/db.collection' using com.mongodb.hadoop.pig.MongoLoader; STORE records INTO 'file:///output.bson' using com.mongodb.hadoop.pig.BSONStorage; MongoDB+Hadoop and PIG
- 44. MongoDB + Hadoop and PIG • PIG has a some special datatypes – Bags – Maps – Tuples • MongoDB+Hadoop Connector converts between PIG and MongoDB datatypes
- 45. raw = LOAD 'hdfs:///messages.bson' using com.mongodb.hadoop.pig.BSONLoader('','headers:[]') ; send_recip = FOREACH raw GENERATE $0#'From' as from, $0#'To' as to; //filter && split send_recip_filtered = FILTER send_recip BY to IS NOT NULL; send_recip_split = FOREACH send_recip_filtered GENERATE from as from, TRIM(FLATTEN(TOKENIZE(to))) as to; //group && count send_recip_grouped = GROUP send_recip_split BY (from, to); send_recip_counted = FOREACH send_recip_grouped GENERATE group, COUNT($1) as count; STORE send_recip_counted INTO 'file:///enron_results.bson' using com.mongodb.hadoop.pig.BSONStorage; PIG
- 46. Upcoming
- 47. Roadmap Features • Performance Improvements–Lazy BSON • Full-Featured Hive Support • Support multi-collection input • API for custom splitter implementations • And lots more …
- 48. Recap • Use Hadoop for massive MapReduce computations on big data sets stored in MongoDB • MongoDB can be used as Hadoop filesystem • There’s lots of tools to make it easier – Streaming – Hive – PIG – EMR • https://github.com/mongodb/mongo-hadoop/tree/ master/examples
- 50. QA?
- 51. MongoDB + Hadoop Senior Solutions Architect, MongoDB Inc. Norberto Leite #bcnmug