![MAP_REDUCE – AN EXAMPLE
Take a look at mocked log extract:
[IP – bandwidth]
10.0.0.1 – 1234
10.0.0.1 – 900
10.0.0.2 – 1230
10.0.0.3 – 999](https://image.slidesharecdn.com/makingbigdatasmall-120713163817-phpapp02/85/Making-Big-Data-small-23-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (7)
Ad
Similar to Making Big Data, small (20)
Ad
Recently uploaded (20)
Making Big Data, small
- 1. MAKING BIG DATA, SMALL Using distributed systems for processing, analysing and managing large huge data sets Marcin Jedyk Software Professional’s Network, Cheshire Datasystems Ltd
- 2. WARM-UP QUESTIONS How many of you heard about Big Data before? How many about NoSQL? Hadoop?
- 3. AGENDA. Intro – motivation, goal and ‘not about…’ What is Big Data? NoSQL and systems classification Hadoop & HDFS MapReduce & live demo HBase
- 4. AGENDA Pig Building Hadoop cluster Conclusions Q&A
- 5. MOTIVATION Data is everywhere – why not to analyse it? With Hadoop and NoSQL systems, building distributed systems is easier than before Relying on software & cheap hardware rather than expensive hardware works better!
- 6. MOTIVATION
- 7. GOAL To explain basic ideas behind Big Data To present different approaches towards BD To show that Big Data systems are easy to build To show you where to start with such systems
- 8. WHAT IT IS NOT ABOUT? Not a detailed lecture on a single system Not about advanced techniques in Big Data Not only about technology – but also about its application
- 9. WHAT IS BIG DATA? Data characterised by 3 Vs: Volume Variety Velocity The interesting ones: variety & velocity
- 10. WHAT IS BIG DATA Data of high velocity: cannot store? Process on the fly! Data of high variety: doesn’t fit into relational schema? Don’t use schema, use NoSQL! Data which is impractical to process on a single server
- 11. NO-SQL Hand in and with Big Data NoSQL – an umbrella term for non-relational data bases or data storages It’s not always possible to replace RDBMS with NoSQL! (opposite is also true)
- 12. NO-SQL NoSQL DBs are built around different principles Key-value stores: Redis, Riak Document stores: i.e. MongoDB – record as a document; each entry has its own meta-data (JSON like, BSON) Table stores: i.e. Hbase – data persisted in multiple columns (even millions), billions of rows and multiple versions of records
- 13. HADOOP Existed before ‘Big Data’ buzzword emerged A simple idea – MapReduce A primary purpose – to crunch tera- and petabytes of data HDFS as underlying distributed file system
- 14. HADOOP – ARCHITECTURE BY EXAMPLE Image you need to process 1TB of logs What would you need? A server!
- 15. HADOOP – ARCHITECTURE BY EXAMPLE But 1TB is quite a lot of data… we want it quicker! Ok, what about distributed environment?
- 16. HADOOP – ARCHITECTURE BY EXAMPLE So what about that Hadoop stuff? Each node can: store data & process it (DataNode & TaskTracker)
- 17. HADOOP – ARCHITECTURE BY EXAMPLE How about allocating jobs to slaves? We need a JobTracker!
- 18. HADOOP – ARCHITECTURE BY EXAMPLE How about HDFS, how data blocks are assembled into files? NameNode does it.
- 19. HADOOP – ARCHITECTURE BY EXAMPLE NameNode – manages HDFS metadata, doesn’t deal with files directly JobTracker – schedules, allocates and monitors job execution on slaves – TaskTrackers TaskTracker – runs MapReduce operations DataNode – stores blocks of HDFS – default replication level for each block: 3
- 20. HADOOP - LIMITATIONS DataNodes & TaskTrackers are fault tollerant NameNode & JobTracker are NOT! (existing workaround for this problem) HDFS deals nicely with large files, doesn’t do well with billions of small files
- 21. MAP_REDUCE MapReduce – parallelisation approach Two main stages: Map – do an actual bit of work, i.e.: extract info Reduce – summarise, aggregate or filter outputs from Map operation For each job, multiple Map and Reduce operations – each may run on different node = parallelism
- 22. MAP_REDUCE – AN EXAMPLE Let’s process 1TB of raw logs and extract traffic by host. After submitting a job, JobTracker allocates tasks to slaves – possibly divided into 64MB packs = 16384 Map operations! Map - analyse logs and return them as set of <key,value> Reduce -> merge output of Map operations
- 23. MAP_REDUCE – AN EXAMPLE Take a look at mocked log extract: [IP – bandwidth] 10.0.0.1 – 1234 10.0.0.1 – 900 10.0.0.2 – 1230 10.0.0.3 – 999
- 24. MAP_REDUCE – AN EXAMPLE It’s important to define key, in this case IP <10.0.0.1;2134> <10.0.0.2;1230> <10.0.0.3;999> Now, assume another Map operation returned: <10.0.0.1;1500> <10.0.0.3;1000> <10.0.0.4;500>
- 25. MAP_REDUCE – AN EXAMPLE Now, Reduce will merge those results: <10.0.0.1;3624> <10.0.0.2;2230> <10.0.0.3;1499> <10.0.0.4;500>
- 26. MAP_REDUCE Selecting a key is important It’s possible to define composite key, i.e. IP+date For more complex tasks, it’s possible to chain MapReduce jobs
- 27. HBASE Another layer on top of Hadoop/HDFS A distributed data storage Not a replacement for RDBMS! Can be used with MapReduce Good for unstructured data – no need to worry about exact schema in advance
- 28. PIG – HBASE ENHANCEMENT HBase - missing proper query language Pig – makes life easier for HBase users Translates queries into MapReduce jobs When working with Pig or HBase, forget what you know about SQL – it makes your life easier
- 29. BUILDING HADOOP CLUSTER Post production servers are ok Don’t take ‘cheap hardware’ too literally Good connection between nodes is a must! >=1Gbps between nodes >=10Gbps between racks 1 disk per CPU core More RAM, more caching!
- 30. FINAL CONCLUSIONS Hadoop and NoSQL-like DB/DS scale very well Hadoop ideal for crunching huge data sets Does very well in production environment Cluster of slaves is fault tolerant, NameNode and JobTracker are not!
- 31. EXTERNAL RESOURCES Trending Topic – build on Wikipedia access logs: http://goo.gl/BWWO1 Building web crawler with Hadoop: http://goo.gl/xPTlJ Analysing adverse drug events: http://goo.gl/HFXAx Moving average for large data sets: http://goo.gl/O4oml
- 32. EXTERNAL RESOURCES – USEFUL LINKS http://www.slideshare.net/fullscreen/jpatanooga/la-hug-dec-2011- recommendation-talk/1 https://ccp.cloudera.com/display/CDH4DOC/CDH4+Installation+Guide http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html http://hstack.org/hbase-performance-testing/ http://www.theregister.co.uk/2012/06/12/hortonworks_data_platform_one/ http://wiki.apache.org/hadoop/MachineScaling http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote- ladis2009.pdf http://www.cloudera.com/resource-types/video/ http://hstack.org/why-were-using-hbase-part-2/
- 33. QUESTIONS?