Faculty OfEngineering& Technology

BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

FACULTY OF ENGINEERING AND TECHNOLOGY

Big Data Analytics (203105348)

7th SEMESTER

7A9 (CSE)

Name: Aashutosh.S.yadav

Year/Sem: 4th

Enrolment No. 2203051057108

Course: B-tech(CSE)

2203051057108 1
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

CERTIFICATE

This is to certify that

Mr./Ms..............................................................................................................
with enrolment no. ................................................................ has
successfully completed his/her laboratory experiments in the Big Data
Analytics (203105348) From the Department of

...................................................................................................

during the academic year ............................................

Date of Submission: -........................... Staff In charge: -...........................

2203051057108 2
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

INDEX

PAGE NO. MARKS SIGN

SR DATE OF DATE OF
NO PRACTICAL LIST START COMPLETION From To
1 To understand the overall 15-0602024 15-06-2024 4 6
programming architecture
using Map Reduce API.
2 Write a program of Word 7 10
Count in Map Reduce over
HDFS.
3
Basic CRUD operations in
MongoDB
4 Store the basic information
about students such as roll
no, name, date of birth, and
address of student using
various collection types
such as List, Set and Map.
5 Basic commands available
for the Hadoop Distributed
File System
6
Basic commands available
for HIVE Query Language.
7 Basic commands of HBASE
Shell.
8 Creating the HDFS tables
and loading them in Hive
and learn joining of tables
in Hive

2203051057108 3
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

Practical:1
AIM: To understand the overall programming architecture using Map Reduce API.

The MapReduce task is mainly divide into into two phase map phase and Reduce Phase.
1. Map(), filter(), and reduce() in python.
2. These functions are most commonly used with lambda function.
1.Map():
“A map function execute certain instructions or functionality provided to it on every item of an
iterable could be a list, tuple, set, etc.
SYNTAX:
Map(function,iterable)
EXAMPLE:
items=[1,2,3,4,5]
a=list(map((lambda x: x **3), items))
print(a)

2.Filter():-
“A filter function in python tests a specific user-defined confition for a function and returns an
iterable for the elements and values that satisfy the condition or, in other words, return true.”

2203051057108 4
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

SYNTAX:
Filter(function, iterable)

EXAMPLE:
a=[1,2,3,4,5]
b=[2,5,0,7,3]
c=list(filter(lambda x: x in a,b))
print(c)# prints out[2,5,3]

3.Reduce():
“Reduce function apply a function to every item of an iterable and gives back a single value as a
resultant”.
We have to import the reduce function from functools module using the statement.
SYNTAX:
reduce(function, iterable)
EXAMPLE:
from functools import reduce
a=reduce((lambda x, y: x*y),[1,2,3,4,])
print(a)

2203051057108 5
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

2203051057108 6
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

Practical-2

Aim: Write a program of Word Count in Map Reduce over HDFS.

Description:
MapReduce is a framework for processing large datasets using a large number of computers
(nodes), collectively referred to as a cluster. Processing can occur on data stored in a file
system (HDFS).A method for distributing computation across multiple nodes.Each node
processes the data that is stored at that node.
Consists of two main phases
Mapper Phase
Reduce phase

Map

I/P File Reduce HDFS

Map

Input data set is split into independent blocks – processed in parallel. Each input split is
converted in Key Value pairs. Mapper logic processes each key value pair and produces and
intermediate key value pairs based on the implementation logic. Resultant key value pairs can
be of different type from that of input key value pairs. The output of Mapper is passed to the
reducer. Output of Mapper function is the input for Reducer. Reducer sorts the intermediate

2203051057108 7
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

key value pairs. Applies reducer logic upon the key value pairs and produces the output in
desired format.Output is stored in HDFS.

The overall MapReduce word count process

Python Code
import urllib.request

import random

from operator import itemgetter

import_word = {}

import_count = 0

story = 'http://sixty-north.com/c/t.txt'

2203051057108 8
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

request =urllib.request.Request(story)

response = urllib.request.urlopen(request)

each_word = []

words = 1

same_words = {}

word = []

""" lopping the entire file"""

#Collect All the words into a list

for line in response:

#print "Line " , line

line_words = line.split()

for word in line_words:

each_word.append(word)

for words in each_word:

if words .lower() not in same_words.keys():

same_words[words.lower()] = 1

else:

same_words[words.lower()] =same_words[words.lower()]= +1

for each in same_words.keys():

print("word = ",each,"count = ",same_words[each]

2203051057108 9
 Faculty OfEngineering& Technology
BIG-DATA ANALYSIS(203105348)
B.Tech CSE 4th Year 7th Semester

Output:-

2203051057108 10