“Hadoop”

A Distributed Architecture, FileSystem, & MapReduce

H. Andrew Schwartz

CSE545
Spring 2023

(freesvg.org/1534373472)
 Big Data Analytics, The Class
Goal: Generalizations
A model or summarization of the data.

Data Workflow Frameworks Analytics and Algorithms

Hadoop File System Similarity Search

Spark Hypothesis Testing
Streaming Transformers/Self-Supervision
MapReduce
Deep Learning Frameworks Recommendation Systems
Link Analysis
Big Data Analytics, The Class

W st
or em
Sy s
kf s
m
ir th
lo
w l go
A

Big Data Analytics

a l

D ols
tic s

is
To
s

tri
i
at hod

bu
t
S et

ted
M
Big Data Analytics, The Class

W st
or em
Sy s
kf s
m
ir th
lo
w l go
A

Big Data Analytics

a l

D ols
tic s

is
To
s

tri
i
at hod

bu
t
S et

ted
M
Data
Classical Data Analytics

CPU

Memory

Disk
Classical Data Analytics

CPU

Memory
(64 GB)

Disk
Classical Data Analytics

CPU

Memory
(64 GB)

Disk
Classical Data Analytics

CPU

Memory
(64 GB)

Disk
 IO Bounded
Reading a word from disk versus main memory: 105 slower!
Reading many contiguously stored words
is faster per word, but fast modern disks
still only reach ~1GB/s for sequential reads.
 IO Bounded
Reading a word from disk versus main memory: 105 slower!
Reading many contiguously stored words
is faster per word, but fast modern disks
still only reach ~1GB/s for sequential reads.

IO Bound: biggest performance bottleneck is reading / writing to disk.

starts around 500 GBs: >10 minutes just to read

500 TBs: ~8,600 minutes = ~6 days
Classical Big Data

CPU
Classical focus: efficient use of disk.
e.g. Apache Lucene / Solr
Memory

Disk Classical limitation: Still bounded when

needing to process all of a large file.
Classical Big Data

Classical focus: efficient use of disk.

How to solve?
e.g. Apache Lucene / Solr

Classical limitation: Still bounded when

needing to process all of a large file.
Distributed Architecture
Switch
~10Gbps

Rack 1
Rack 2
Switch Switch
~1Gbps ~1Gbps
...

CPU CPU CPU CPU CPU CPU

Memory Memory ... Memory Memory Memory ... Memory

Disk Disk Disk Disk Disk Disk

 Distributed Architecture
In reality, modern setups often have multiple cpus and disks
per server, but we will model as if one machine
per cpu-disk pair.
Switch
~1Gbps

CPU CPU CPU CPU CPU CPU

... ...
...
Memory Memory

Disk Disk ... Disk Disk Disk ... Disk

Distributed Architecture (Cluster)

Switch
~10Gbps

Rack 1
Rack 2
Switch Switch
~1Gbps ~1Gbps
...

CPU CPU CPU CPU CPU CPU

Memory Memory ... Memory Memory Memory ... Memory

Disk Disk Disk Disk Disk Disk

 Distributed Architecture (Cluster)
Challenges for IO Cluster Computing

1. Nodes fail
1 in 1000 nodes fail a day

2. Network is a bottleneck
Typically 1-10 Gb/s throughput

3. Traditional distributed programming is

often ad-hoc and complicated
 Distributed Architecture (Cluster)
Challenges for IO Cluster Computing
1. Nodes fail
1 in 1000 nodes fail a day
Duplicate Data
2. Network is a bottleneck
Typically 1-10 Gb/s throughput
Bring computation to nodes, rather than
data to nodes.
3. Traditional distributed programming is
often ad-hoc and complicated
Stipulate a programming system that
can easily be distributed
 Distributed Architecture (Cluster)
Challenges for IO Cluster Computing
1. Nodes fail
1 in 1000 nodes fail a day
Duplicate Data
2. Network is a bottleneck
Typically 1-10 Gb/s throughput HDFS with
Bring computation to nodes, rather than MapReduce
data to nodes. accomplishes all!
3. Traditional distributed programming is
often ad-hoc and complicated
Stipulate a programming system that
can easily be distributed
Distributed Filesystem

The effectiveness of MapReduce, Spark, and other

distributed processing systems is in part simply due to
use of a distributed filesystem!
Distributed Filesystem
Characteristics for Big Data Tasks
Large files (i.e. >100 GB to TBs)
Reads are most common
No need to update in place
(append preferred)
CPU

Memory

Disk
 Distributed Filesystem
(e.g. Apache HadoopDFS, GoogleFS, EMRFS)

C, D: Two different files

https://opensource.com/life/14/8/intro
-apache-hadoop-big-data

C
D
 Distributed Filesystem
“Hadoop” was named after a
toy elephant belonging to Doug
Cutting’s son. Cutting was one
(e.g. Apache HadoopDFS, GoogleFS, EMRFS)
of Hadoop’s creators.
C, D: Two different files

https://opensource.com/life/14/8/intro
-apache-hadoop-big-data

C
D
 Distributed Filesystem
(e.g. Apache HadoopDFS, GoogleFS, EMRFS)

C, D: Two different files; break into chunks (or "partitions"):

C0 D0

C1 D1

C2 D2

C3 D3

C4 D4

C5 D5
 Distributed Filesystem
(e.g. Apache HadoopDFS, GoogleFS, EMRFS)

C, D: Two different files

chunk server 1 chunk server 2 chunk server 3 chunk server n

(Leskovec at al., 2014; http://www.mmds.org/)

 Distributed Filesystem
(e.g. Apache HadoopDFS, GoogleFS, EMRFS)

C, D: Two different files

chunk server 1 chunk server 2 chunk server 3 chunk server n

(Leskovec at al., 2014; http://www.mmds.org/)

 Distributed Filesystem
(e.g. Apache HadoopDFS, GoogleFS, EMRFS)

C, D: Two different files

chunk server 1 chunk server 2 chunk server 3 chunk server n

(Leskovec at al., 2014; http://www.mmds.org/)

 Distributed Filesystem
Chunk servers (on Data Nodes)
File is split into contiguous chunks
Typically each chunk is 16-64MB
Each chunk replicated (usually 2x or 3x)
Try to keep replicas in different racks

(Leskovec at al., 2014; http://www.mmds.org/)

 Components of a Distributed Filesystem
Chunk servers (on Data Nodes)
File is split into contiguous chunks
Typically each chunk is 16-64MB
Each chunk replicated (usually 2x or 3x)
Try to keep replicas in different racks
Name node (aka master node)
Stores metadata about where files are stored
Might be replicated or distributed across data nodes.

(Leskovec at al., 2014; http://www.mmds.org/)

 Components of a Distributed Filesystem
Chunk servers (on Data Nodes)
File is split into contiguous chunks
Typically each chunk is 16-64MB
Each chunk replicated (usually 2x or 3x)
Try to keep replicas in different racks
Name node (aka master node)
Stores metadata about where files are stored
Might be replicated or distributed across data nodes.
Client library for file access
Talks to master to find chunk servers
Connects directly to chunk servers to access data

(Leskovec at al., 2014; http://www.mmds.org/)

 Distributed Architecture (Cluster)
Challenges for IO Cluster Computing
1. Nodes fail
1 in 1000 nodes fail a day
Duplicate Data (Distributed FS)
2. Network is a bottleneck
Typically 1-10 Gb/s throughput
Bring computation to nodes, rather than
data to nodes.
3. Traditional distributed programming is
often ad-hoc and complicated
Stipulate a programming system that
can easily be distributed
 What is MapReduce
noun.1 - A style of programming

input chunks => map tasks | group_by keys | reduce tasks => output

“|” is the linux “pipe” symbol: passes stdout from first process to stdin of next.
 What is MapReduce
noun.1 - A style of programming

input chunks => map tasks | group_by keys | reduce tasks => output

“|” is the linux “pipe” symbol: passes stdout from first process to stdin of next.

E.g. counting words:

tokenize(document) | sort | uniq -c

 What is MapReduce
noun.1 - A style of programming

input chunks | map tasks | group_by keys | reduce tasks => output

“|” is the linux “pipe” symbol: passes output from first process to input of next.

E.g. counting words:

cat file.txt | tr -s '[[:space:]]' '\n' | sort | uniq -c

noun.2 - A system that distributes MapReduce style programs across a

distributed file-system.

(e.g. Google’s internal “MapReduce” or apache.hadoop.mapreduce with hdfs)

 What is MapReduce
noun.1 - A style of programming

input chunks => map tasks | group_by keys | reduce tasks => output

“|” is the linux “pipe” symbol: passes output from first process to input of next.

E.g. counting words:

tokenize(document) | sort | uniq -c

noun.2 - A system that distributes MapReduce style programs across a

distributed file-system.

(e.g. Google’s internal “MapReduce” or apache.hadoop.mapreduce with hdfs)

What is MapReduce
 What is MapReduce

extract what
you care
about.

line => (k, v) Map

 What is MapReduce

sort and
shuffle

many (k, v) =>

(k, [v1, v2]), ...
extract what
you care
about.

Map
 What is MapReduce

sort and
shuffle
extract what
you care
about. aggregate,
summarize
Map
Reduce
 What is MapReduce
Easy as 1, 2, 3!
Step 1: Map Step 2: Sort / Group by Step 3: Reduce
 What is MapReduce
Easy as 1, 2, 3!
Step 1: Map Step 2: Sort / Group by Step 3: Reduce

(Leskovec at al., 2014; http://www.mmds.org/)

(1) The Map Step

(Leskovec at al., 2014; http://www.mmds.org/)

 (2) The Sort / Group-by Step

(Leskovec at al., 2014; http://www.mmds.org/)

 (3) The Reduce Step

(Leskovec at al., 2014; http://www.mmds.org/)

 What is MapReduce
Easy as 1, 2, 3!
Step 1: Map Step 2: Sort / Group by Step 3: Reduce

(Leskovec at al., 2014; http://www.mmds.org/)

What is MapReduce
Map: (k,v) -> (k’, v’)*
(Written by programmer)

Group by key: (k1’, v1’), (k2’, v2’), ... -> (k1’, (v1’, v’, …),
(system handles) (k2’, (v1’, v’, …), …

Reduce: (k’, (v1’, v’, …)) -> (k’, v’’)*

(Written by programmer)
Example: Word Count
tokenize(document) | sort | uniq -c
Example: Word Count
tokenize(document) | sort | uniq -c

Map: extract
what you sort and Reduce:
care about. shuffle aggregate,
summarize
Example: Word Count

(Leskovec at al., 2014; http://www.mmds.org/)

(Leskovec at al., 2014;
http://www.mmds.org/)

Chunks
 Example: Word Count
@abstractmethod
def map(k, v):
pass

@abstractmethod
def reduce(k, vs):
pass
 Example: Word Count (v1)
def map(k, v):
for w in tokenize(v):
yield (w,1)

def reduce(k, vs):

return len(vs)
 Example: Word Count (v1)
def map(k, v): def tokenize(s):
for w in tokenize(v): #simple version
yield (w,1) return s.split(‘ ‘)

def reduce(k, vs):

return len(vs)
 Example: Word Count (v2)
def map(k, v):
counts = dict()
for w in tokenize(v):

counts each word within the chunk

(try/except is faster than
“if w in counts”)
 Example: Word Count (v2)
def map(k, v):
counts = dict()
for w in tokenize(v):
try:
counts[w] += 1 counts each word within the chunk
except KeyError: (try/except is faster than
counts[w] = 1 “if w in counts”)
for item in counts.iteritems():
yield item
 Example: Word Count (v2)
def map(k, v):
counts = dict()
for w in tokenize(v):
try:
counts[w] += 1 counts each word within the chunk
except KeyError: (try/except is faster than
counts[w] = 1 “if w in counts”)
for item in counts.iteritems():
yield item

def reduce(k, vs): sum of counts from different chunks

return (k, sum(vs) )
 Distributed Architecture (Cluster)
Challenges for IO Cluster Computing
1. Nodes fail
1 in 1000 nodes fail a day
Duplicate Data (Distributed FS)
2. Network is a bottleneck
Typically 1-10 Gb/s throughput
Bring computation to nodes, rather than
data to nodes.
3. Traditional distributed programming is
often ad-hoc and complicated
Stipulate a programming system that
can easily be distributed
 Distributed Architecture (Cluster)
Challenges for IO Cluster Computing
1. Nodes fail
1 in 1000 nodes fail a day
Duplicate Data (Distributed FS)
2. Network is a bottleneck
Typically 1-10 Gb/s throughput
Bring computation to nodes, rather than
data to nodes. (Sort and Shuffle)
3. Traditional distributed programming is
often ad-hoc and complicated
Stipulate a programming system that
can easily be distributed
 Distributed Architecture (Cluster)
Challenges for IO Cluster Computing
1. Nodes fail
1 in 1000 nodes fail a day
Duplicate Data (Distributed FS)
2. Network is a bottleneck
Typically 1-10 Gb/s throughput
Bring computation to nodes, rather than
data to nodes. (Sort and Shuffle)
3. Traditional distributed programming is
often ad-hoc and complicated (Simply define a map
Stipulate a programming system that and reduce)
can easily be distributed
 Example: Relational Algebra

Select

Project

Union, Intersection, Difference

Natural Join

Grouping
 Example: Relational Algebra

Select

Project

Union, Intersection, Difference

Natural Join

Grouping
 Example: Relational Algebra

Select

R(A1,A2,A3,...), Relation R, Attributes A*

return only those attribute tuples where condition C is true

 Example: Relational Algebra
Select
R(A1,A2,A3,...), Relation R, Attributes A*
return only those attribute tuples where condition C is true
def map(k, v): #v is list of attribute tuples: [(...,), (...,), ...]
r = []
for t in v:
if t satisfies C:
r += [(t, t)]
return r
 Example: Relational Algebra
Select
R(A1,A2,A3,...), Relation R, Attributes A*
return only those attribute tuples where condition C is true
def map(k, v): #v is list of attribute tuples: [(...,), (...,), ...]
r = []
for t in v:
if t satisfies C:
r += [(t, t)]
return r
def reduce(k, vs):
r = []
for each v in vs:
r += [(k, v)]
return r
 Example: Relational Algebra

Select

R(A1,A2,A3,...), Relation R, Attributes A*

return only those attribute tuples where condition C is true

def map(k, v): #v is list of attribute tuples
for t in v:
if t satisfies C:
yield (t, t)

def reduce(k, vs):

For each v in vs:
yield (k, v)
 Example: Relational Algebra
Natural Join
Given R1 and R2 return Rjoin
-- union of all pairs of tuples that match given attributes.
def map(k, v): #k \in {R1, R2}, v is (A, B) for R1, (B, C) for R2
#B are matched attributes
 Example: Relational Algebra
Natural Join
Given R1 and R2 return Rjoin
-- union of all pairs of tuples that match given attributes.
def map(k, v): #k \in {R1, R2}, v is (A, B) for R1, (B, C) for R2
#B are matched attributes
if k==’R1’:
(a, b) = v
return (b,(‘R1’,a))
if k==’R2’:
(b,c) = v
return (b,(‘R2’,c))
 Example: Relational Algebra
Natural Join
Given R1 and R2 return Rjoin
-- union of all pairs of tuples that match given attributes.
def map(k, v): #k \in {R1, R2}, v is (A, B) for R1, (B, C) for R2
#B are matched attributes
if k==’R1’:
def reduce(k, vs):
(a, b) = v
return (b,(‘R1’,a)) r1, r2, rjn = [], [], []
if k==’R2’: for (s, x) in vs: #separate rs
(b,c) = v if s == ‘R1’: r1.append(x)
return (b,(‘R2’,c)) else: r2.append(x)
for a in r1: #join as tuple
for each c in r2:
rjn += (‘Rjoin’, (a, k, c)) #k is b
return rjn
Data Flow
Data Flow

hash

(Leskovec at al., 2014; http://www.mmds.org/)

Data Flow

Programmer

hash

Programmer

(Leskovec at al., 2014; http://www.mmds.org/)

Data Flow

DFS Map Map’s Local FS Reduce DFS

Data Flow

MapReduce system handles:

● Partitioning
● Scheduling map / reducer execution
● Group by key

● Restarts from node failures

● Inter-machine communication
Data Flow

DFS MapReduce DFS

● Schedule map tasks near physical storage of chunk

● Intermediate results stored locally
● Master / Name Node coordinates
Data Flow

DFS MapReduce DFS

● Schedule map tasks near physical storage of chunk

● Intermediate results stored locally
● Master / Name Node coordinates
○ Task status: idle, in-progress, complete
○ Receives location of intermediate results and schedules with reducer
○ Checks nodes for failures and restarts when necessary
■ All map tasks on nodes must be completely restarted
■ Reduce tasks can pickup with reduce task failed
Data Flow

DFS MapReduce DFS

● Schedule map tasks near physical storage of chunk

● Intermediate results stored locally
● Master / Name Node coordinates
○ Task status: idle, in-progress, complete
○ Receives location of intermediate results and schedules with reducer
○ Checks nodes for failures and restarts when necessary
■ All map tasks on nodes must be completely restarted
■ Reduce tasks can pickup with reduce task failed

DFS MapReduce DFS MapReduce DFS

Data Flow

Skew: The degree to which certain tasks end up taking much

longer than others.

Handled with:

● More reducers (i.e. partitions) than reduce tasks

● More reduce tasks than nodes
Data Flow

Key Question: How many Map and Reduce jobs?

M: map tasks, R: reducer tasks
Data Flow

Key Question: How many Map and Reduce jobs?

M: map tasks, R: reducer tasks
CPU CPU CPU

Answer: 1) If possible, one chunk per map task Mem Mem . Mem
(maximizes flexibility for scheduling) .
.
Disk Disk Disk
2) M >> |nodes| ≈≈ |cores|
(better handling of node failures, better load balancing)
3) R <= M
(reduces number of parts stored in DFS)
Data Flow Tasks (Map Task or Reduce Task)
version 1: few reduce tasks
(same number of reduce tasks as nodes)

node1

node2

node3

node4

node5

time
tasks represented by
time to complete task
(some tasks take much longer)
Data Flow Tasks (Map Task or Reduce Task)
version 1: few reduce tasks version 2: more reduce tasks
(same number of reduce tasks as nodes) (more reduce tasks than nodes)

node1 node1

node2 node2

node3 node3

node4 node4

node5 node5

time time
tasks represented by tasks represented by
time to complete task time to complete task
(some tasks take much longer) (some tasks take much longer)
Data Flow Tasks (Map Task or Reduce Task)
version 1: few reduce tasks version 2: more reduce tasks
(same number of reduce tasks as nodes) (more reduce tasks than nodes)

node1 node1 node1

Last task
completed
node2 node2 node2

node3 node3 Can node3

redistribute
these tasks to
node4 node4 other nodes node4

node5 node5 node5

time time time

tasks represented by tasks represented by
time to complete task time to complete task (the last task now completes
(some tasks take much longer) (some tasks take much longer) much earlier )
 Communication Cost Model

How to assess performance?

(1) Computation: Map + Reduce + System Tasks

(2) Communication: Moving (key, value) pairs

 Communication Cost Model

How to assess performance?

(1) Computation: Map + Reduce + System Tasks

(2) Communication: Moving (key, value) pairs

Ultimate Goal: wall-clock Time.

 Communication Cost Model

How to assess performance?

(1) Computation: Map + Reduce + System Tasks

● Mappers and reducers often single pass O(n) within node
(2) Communication: Moving
● System: sort the keys key,
is usually value
most pairs
expensive
● Even if map executes on same node, disk read usually
dominates
● In any case, can add more nodes
Ultimate Goal: wall-clock Time.
 Communication Cost Model

How to assess performance?

(1) Computation: Map + Reduce + System Tasks

(2) Communication: Moving key, value pairs

Often dominates computation.
● Connection speeds: 1-10 gigabits per sec;
Ultimate HD
Goal:
read:wall-clock Time.
50-150 gigabytes per sec
● Even reading from disk to memory typically takes longer than
operating on the data.
 Communication Cost Model

How to assess performance?

Communication
(1) Cost Map
Computation: = input size +
+ Reduce + System Tasks
(sum of size of all map-to-reducer files)

(2) Communication: Moving key, value pairs

Often dominates computation.
● Connection speeds: 1-10 gigabits per sec;
Ultimate HD
Goal:
read:wall-clock Time.
50-150 gigabytes per sec
● Even reading from disk to memory typically takes longer than
operating on the data.
 Communication Cost Model

How to assess performance?

Communication
(1) Cost Map
Computation: = input size +
+ Reduce + System Tasks
(sum of size of all map-to-reducer files)

(2) Communication: Moving key, value pairs

Often dominates computation.
● Connection speeds: 1-10 gigabits per sec;
UltimateHDGoal:
read:wall-clock Time.
50-150 gigabytes per sec
● Even reading from disk to memory typically takes longer than
operating on the data.
● Output from reducer ignored because it’s either small (finished
summarizing data) or being passed to another mapreduce job.
 Communication Cost: Natural Join

R, S: Relations (Tables) R(A, B) ⨝ S(B, C)

Communication Cost = input size +

(sum of size of all map-to-reducer files)

DFS Map LocalFS Network Reduce DFS ?

(Anytime where MapReduce would need to write and read from disk a lot).
 Communication Cost: Natural Join
R, S: Relations (Tables) R(A, B) ⨝ S(B, C)

Communication Cost = input size +

(sum of size of all map-to-reducer files)

def reduce(k, vs):

r1, r2 = [], []
def map(k, v): for (rel, x) in vs: #separate rs
if k==”R1”: if rel == ‘R’: r1.append(x)
(a, b) = v else: r2.append(x)
yield (b,(R1,a))
for a in r1: #join as tuple
if k==”R2”:
(b,c) = v for each c in r2:
yield (b,(R2,c)) yield (Rjoin’, (a, k, c)) #k is
b
 Communication Cost: Natural Join
R, S: Relations (Tables) R(A, B) ⨝ S(B, C)

Communication Cost = input size +

(sum of size of all map-to-reducer files)

= |R1| + |R2| + (|R1| + |R2|)

def reduce(k, vs):
= O(|R1| + |R2|)
r1, r2 = [], []
def map(k, v): for (rel, x) in vs: #separate rs
if k==”R1”: if rel == ‘R’: r1.append(x)
(a, b) = v else: r2.append(x)
yield (b,(R1,a))
for a in r1: #join as tuple
if k==”R2”:
(b,c) = v for each c in r2:
yield (b,(R2,c)) yield (Rjoin’, (a, k, c)) #k is
b
MapReduce: Final Considerations
● Performance Refinements:
○ Combiners (like word count version 2 but done via reduce)
■ Run reduce right after map from same node before passing to
reduce (MapTask can execute)
■ Reduces communication cost

○ Backup tasks (aka speculative tasks)

■ Schedule multiple copies of tasks when close to the end to
mitigate certain nodes running slow.

○ Override partition hash function to organize data

E.g. instead of hash(url) use hash(hostname(url))
MapReduce: Final Considerations
● Performance Refinements:
○ Combiners (like word count version 2 but done via reduce)
■ Run reduce right after map from same node before passing to
reduce (MapTask can execute)
■ Reduces communication cost but requires commutative
reduce steps
○ Backup tasks (aka speculative tasks)
■ Schedule multiple copies of tasks when close to the end to
mitigate certain nodes running slow.

○ Override partition hash function to organize data

E.g. instead of hash(url) use hash(hostname(url))
MapReduce: Final Considerations
● Performance Refinements:
○ Combiners (like word count version 2 but done via reduce)
■ Run reduce right after map from same node before passing to
reduce (MapTask can execute)
■ Reduces communication cost but requires commutative
reduce steps
○ Backup tasks (aka speculative tasks)
■ Schedule multiple copies of tasks when close to the end to
mitigate certain nodes running slow.

○ Override partition hash function to organize data

E.g. instead of hash(url) use hash(hostname(url))