Map Reduce

Lecture 2 – MapReduce CPE 458 – Parallel Programming, Spring 2009 Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution 2.5 License.http://creativecommons.org/licenses/by/2.5

Outline MapReduce: Programming Model MapReduce Examples A Brief History MapReduce Execution Overview Hadoop MapReduce Resources

MapReduce “ A simple and powerful interface that enables automatic parallelization and distribution of large-scale computations, combined with an implementation of this interface that achieves high performance on large clusters of commodity PCs.” Dean and Ghermawat, “MapReduce: Simplified Data Processing on Large Clusters”, Google Inc.

MapReduce More simply, MapReduce is: A parallel programming model and associated implementation.

Programming Model Description The mental model the programmer has about the detailed execution of their application. Purpose Improve programmer productivity Evaluation Expressibility Simplicity Performance

Programming Models von Neumann model Execute a stream of instructions (machine code) Instructions can specify Arithmetic operations Data addresses Next instruction to execute Complexity Track billions of data locations and millions of instructions Manage with: Modular design High-level programming languages (isomorphic)

Programming Models Parallel Programming Models Message passing Independent tasks encapsulating local data Tasks interact by exchanging messages Shared memory Tasks share a common address space Tasks interact by reading and writing this space asynchronously Data parallelization Tasks execute a sequence of independent operations Data usually evenly partitioned across tasks Also referred to as “Embarrassingly parallel”

MapReduce: Programming Model Process data using special map () and reduce () functions The map() function is called on every item in the input and emits a series of intermediate key/value pairs All values associated with a given key are grouped together The reduce() function is called on every unique key, and its value list, and emits a value that is added to the output

MapReduce: Programming Model How now Brown cow How does It work now brown 1 cow 1 does 1 How 2 it 1 now 2 work 1 M M M M R R <How,1> <now,1> <brown,1> <cow,1> <How,1> <does,1> <it,1> <work,1> <now,1> <How,1 1> <now,1 1> <brown,1> <cow,1> <does,1> <it,1> <work,1> Input Output Map Reduce MapReduce Framework

MapReduce: Programming Model More formally, Map(k1,v1) --> list(k2,v2) Reduce(k2, list(v2)) --> list(v2)

MapReduce Runtime System Partitions input data Schedules execution across a set of machines Handles machine failure Manages interprocess communication

MapReduce Benefits Greatly reduces parallel programming complexity Reduces synchronization complexity Automatically partitions data Provides failure transparency Handles load balancing Practical Approximately 1000 Google MapReduce jobs run everyday.

MapReduce Examples Word frequency Map doc Reduce <word,3> <word,1> <word,1> <word,1> Runtime System <word,1,1,1>

MapReduce Examples Distributed grep Map function emits <word, line_number> if word matches search criteria Reduce function is the identity function URL access frequency Map function processes web logs, emits <url, 1> Reduce function sums values and emits <url, total>

A Brief History Functional programming (e.g., Lisp) map() function Applies a function to each value of a sequence reduce() function Combines all elements of a sequence using a binary operator

MapReduce Execution Overview The user program, via the MapReduce library, shards the input data User Program Input Data Shard 0 Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 * Shards are typically 16-64mb in size

MapReduce Execution Overview The user program creates process copies distributed on a machine cluster. One copy will be the “Master” and the others will be worker threads. User Program Master Workers Workers Workers Workers Workers

MapReduce Resources The master distributes M map and R reduce tasks to idle workers. M == number of shards R == the intermediate key space is divided into R parts Master Idle Worker Message(Do_map_task)

MapReduce Resources Each map-task worker reads assigned input shard and outputs intermediate key/value pairs. Output buffered in RAM. Map worker Shard 0 Key/value pairs

MapReduce Execution Overview Each worker flushes intermediate values, partitioned into R regions, to disk and notifies the Master process. Master Map worker Disk locations Local Storage

MapReduce Execution Overview Master process gives disk locations to an available reduce-task worker who reads all associated intermediate data. Master Reduce worker Disk locations remote Storage

MapReduce Execution Overview Each reduce-task worker sorts its intermediate data. Calls the reduce function, passing in unique keys and associated key values. Reduce function output appended to reduce-task’s partition output file. Reduce worker Sorts data Partition Output file

MapReduce Execution Overview Master process wakes up user process when all tasks have completed. Output contained in R output files. wakeup User Program Master Output files

MapReduce Execution Overview Fault Tolerance Master process periodically pings workers Map-task failure Re-execute All output was stored locally Reduce-task failure Only re-execute partially completed tasks All output stored in the global file system

Hadoop Open source MapReduce implementation http: //hadoop .apache.org/core/index.html Uses Hadoop Distributed Filesytem (HDFS) http: //hadoop .apache. org/core/docs/current/hdfs_design .html Java ssh

References Introduction to Parallel Programming and MapReduce, Google Code University http://code.google.com/edu/parallel/mapreduce-tutorial.html Distributed Systems http://code. google . com/edu/parallel/index .html MapReduce: Simplified Data Processing on Large Clusters http://labs. google . com/papers/mapreduce .html Hadoop http: //hadoop .apache.org/core/

Map Reduce

Recommended

More Related Content

What's hot (20)

Similar to Map Reduce (20)

More from Sri Prasanna (20)

Recently uploaded (20)

Map Reduce