Scribd
Upload
7 views

BigData-Assignment3-CSP 554

Uploaded by

emile.mondon.r
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
7 views

BigData-Assignment3-CSP 554

Uploaded by

emile.mondon.r
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

CSP 554 – Big Data Technologies : Assignment 3

1. HDFS uses replication to ensure data availability and reliability, by storing multiple copies of
data blocks across different nodes. Moreover, this also improves fault tolerance and throughput.
However, it works thanks to an adjustment dynamic based on data popularity, which can lead to
uneven data distribution. This will result in “hot spots” where some nodes are overutilized, while
others are underutilized. To improve this data distribution, authors proposed WBRD (Workload-
Aware Balanced Replica Deletion), but it didn’t work as expected. It led to inefficiencies in clusters
with different hardware configurations because it doesn’t account for node heterogeneity. This
heterogeneity is provided by the HaRD. To do that, it evaluates each node’s processing power
based on the number of containers they are running simultaneously. It permits the distributed
replicas to minimize the data transfer overhead, it also improves data locality and finally, it
improves performance.

The paper presents also a lot of experimental evaluations on a 23-node Hadoop cluster, by
comparing HaRD’s performance to default Hadoop and WBRD. We can see that HaRD reduces
average execution time by up to 60.3% compared to default Hadoop and 17% compared to WBRD.
The experiences were measured with reading-intensive, write-intensive, and network-intensive
tasks. Moreover, when there are a lot of users simultaneously, HaRD is also better by improving
performance with an execution time reduced by up to 60% compared to Hadoop. Finally, HaRD
improves data locality, which results in a smaller usage of the network bandwidth, compared to
Hadoop and WBRD (it showed a 6.9% lower network utilization), and it introduces negligible
overhead: the time required for replica deletion operations is smaller (on the order of
milliseconds).

To conclude, HaRD performs to delete replicas, it dynamically adapts itself to changes in node
capabilities and can be installed very easily.
2. Copy of the program WordCount2.py :

from mrjob.job import MRJob

import re

WORD_RE = re.compile(r"[\w']+")

class MRWordCount(MRJob):

def mapper(self, _, line):

for word in WORD_RE.findall(line):

first_letter = word[0]

if 'a' <= first_letter <= 'n':

yield 'a_to_n', 1

else:

yield 'other', 1

def combiner(self, word, counts):

yield word, sum(counts)

def reducer(self, word, counts):

yield word, sum(counts)

if __name__ == '__main__':

MRWordCount.run()
Screenshot of the output :
3. Copy of the program Salaries2.py :

from mrjob.job import MRJob

class MRSalaries(MRJob):

def mapper(self, _, line):

(name,jobTitle,agencyID,agency,hireDate,annualSalary,grossPay) = line.split('\t')

annualSalary = float(annualSalary)

if annualSalary >= 100000.00:

yield 'High', 1

elif 50000 <= annualSalary <= 99999.99:

yield 'Medium', 1

else:

yield 'Low', 1

def combiner(self, salary_cat, counts):

yield salary_cat, sum(counts)

def reducer(self, salary_cat, counts):

yield salary_cat, sum(counts)

if __name__ == '__main__':

MRSalaries.run()
Screenshot of the output :
4. Copy of the program movie.py :

from mrjob.job import MRJob

class MRUserMovieReview(MRJob):

def mapper(self, _, line):

(user_id, movie_id, rating, timestamp) = line.split(',')

yield user_id, 1

def combiner(self, user_id, counts):

yield user_id, sum(counts)

def reducer(self, user_id, counts):

yield user_id, sum(counts)

if __name__ == '__main__':

MRUserMovieReview.run()
Screenshot of the output :

You might also like