Software Requirements:

1. Hadoop : https://hadoop.apache.org/release/2.7.6.html
2. Java : https://www.oracle.com/java/technologies/javase/javase8u211-later-archivedownloads.
html
3. Eclipse : https://www.eclipse.org/downloads/
List of Experiments:
Experiment 1: Week 1, 2:
1. Implement the following Data structures in Java
a) Linked Lists b) Stacks c) Queues d) Set e) Map
Experiment 2: Week 3:
2. (i)Perform setting up and Installing Hadoop in its three operating modes: Standalone, Pseudo distributed, Fully
distributed
(ii)Use web based tools to monitor your Hadoop setup.
Experiment 3: Week 4:
3.Implement the following file management tasks in Hadoop:
􀁸 Adding files and directories
􀁸 Retrieving files
􀁸 Deleting files
Hint: A typical Hadoop workflow creates data files (such as log files) elsewhere and copies them into
HDFS using one of the above command line utilities.
Experiment 4: Week 5:
4. Run a basic Word Count MapReduce program to understand MapReduce Paradigm.
Experiment 5: Week 6:
5. Write a map reduce program that mines weather data.
Weather sensors collecting data every hour at many locations across the globe gather a large volume
of log data, which is a good candidate for analysis with Map Reduce, since it is semi
structured and record-oriented.
Experiment 6: Week 7:
6.Use MapReduce to find the shortest path between two people in a social graph.
Hint: Use an adjacency list to model a graph, and for each node store the distance from the original
node, as well as a back pointer to the original node. Use the mappers to propagate the distance to the
original node, and the reducer to restore the state of the graph. Iterate until the target node has been
reached.
Experiment 7: Week 8:
7. Implement Friends-of-friends algorithm in MapReduce.
Hint: Two MapReduce jobs are required to calculate the FoFs for each user in a social network .The
first job calculates the common friends for each user, and the second job sorts the common friends
by the number of connections to your friends.
Experiment 8: Week 9:
8. Implement an iterative PageRank graph algorithm in MapReduce.
Hint: PageRank can be implemented by iterating a MapReduce job until the graph has converged. The
mappers are responsible for propagating node PageRank values to their adjacent nodes, and the
reducers are responsible for calculating new PageRank values for each node, and for re-creating the
original graph with the updated PageRank values.
Experiment 9: Week 10:
9. Perform an efficient semi-join in MapReduce.
Hint: Perform a semi-join by having the mappers load a Bloom filter from the Distributed Cache,
and then filter results from the actual MapReduce data source by performing membership queries
against the Bloom filter to determine which data source records should be emitted to the reducers.
Experiment 10: Week 11:
10. Install and Run Pig then write Pig Latin scripts to sort, group, join, project, and filter your
data.
Experiment 12: Week 12:
11. Install and Run Hive then use Hive to create, alter, and drop databases, tables, views,
functions, and indexes

Experiment 2: Week 3:

2. (i)Perform setting up and Installing Hadoop in its three operating modes:

Standalone, Pseudo distributed, Fully distributed
(ii)Use web based tools to monitor your Hadoop setup.
Program – 1

AIM: Perform setting up and install HADOOP in its two operating modes, Pseudo-distributed and fully distributed.
About HADOOP
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets
across clusters of computers using simple programming models. It is designed to scale up from single servers
to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver
high-availability, the library itself is designed to detect and handle failures at the application layer, so
delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Setting up HADOOP
Pre-requisites:
1. Java
2. SSH
Before any other steps, we need to set the java environment variable, this can be done in windows from the system
variables window or on linux by adding the following to the variables
file:
export JAVA_HOME=/usr/java/latest
Download and extract the HADOOP binaries
1. wget http://apache.claz.org/hadoop/common/hadoop-3.1.2/
2. hadoop-3.1.2.tar.gz
3. tar xzf hadoop-3.1.2.tar.gz
4. hadoop-3.1.2/* to hadoop/

Pseudo-distributed mode
1. Add the following variables to the system variable file
1 export HADOOP_HOME=/usr/local/hadoop
2 export HADOOP_MAPRED_HOME=$HADOOP_HOME
3 export HADOOP_COMMON_HOME=$HADOOP_HOME
4 export HADOOP_HDFS_HOME=$HADOOP_HOME
5 export YARN_HOME=$HADOOP_HOME
6 export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
7 export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
8 export HADOOP_INSTALL=$HADOOP_HOME
2. Configure HADOOP files
a) Change to the Hadoop directory/etc/Hadoop
b) b. Add the following to the hadoop-env.sh file
c) export JAVA_HOME=/usr/local/jdk1.7.0_71
d) c. Edit the following config files
Program – 1

core-site.xml
1 <configuration>
2 <property>
3 <name>fs.default.name</name>
4 <value>hdfs://localhost:9000</value>
5 </property>
6 </configuration>

hdfs-site.xml

1 <configuration>
2 <property>
3 <name>dfs.replication</name>
4 <value>1</value>
5 </property>
6 <property>
7 <name>dfs.name.dir</name>
8 <value>file:///home/<user_name>/hadoopinfra/hdfs/namenode
9 </value>
10 </property>
11 <property>
12 <name>dfs.data.dir</name>
13 <value>file:///home/<user_name>/hadoopinfra/hdfs/datanode
14 </value>
15 </property>
16 </configuration>
yarn-site.xml

1 <configuration>
2 <property>
3 <name>yarn.nodemanager.aux-services</name>
4 <value>mapreduce_shuffle</value>
5 </property>
6 </configuration>
mapred-site.xml
1 <configuration>
2 <property>
3 <name>mapreduce.framework.name</name>
4 <value>yarn</value>
5 </property>
6 </configuration>

d. Verifying the installation

i. Formatting the namenodes

Program – 1
Fully distributed mode

1 Configure system and create host files on each node

a. For each node, edit eh /etc/hosts/ file and add the IP addresses of the servers e.g.

i. 92.0.2.1 node-master
ii. 192.0.2.2 node1
iii. 192.0.2.3 node2
2. Distribute the authentication key-pairs to the users

a. Login to the node-master and generate ssh-keys

b. Copy the keys tot the other nodes.

3. Download and extract the HADOOP binaries

4. Set the environment variables (same as pseudo-distributed)
5. Edit the core-site.xml file to set NameNode location

1 <?xml version="1.0" encoding="UTF-8"?>

2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
3 <configuration>
4 <property>
5 <name>fs.default.name</name>
6 <value>hdfs://node-master:9000</value>
7 </property>
8 </configuration>

6. Set the HDFS Paths in hdfs-site.xml

1. <configuration>
2. <property>
3. <name>dfs.namenode.name.dir</name>
4. <value>/home/hadoop/data/nameNode</value>
5. </property>
6. <property>
7. <name>dfs.datanode.data.dir</name>
8. <value>/home/hadoop/data/dataNode</value>
9. </property>
10. <property>
11. <name>dfs.replication</name>
12. <value>1</value>
13. </property>
14. </configuration>
7. Set the Job scheduler (same as pseudo-distributed)
8. Configure YARN in yarn-site.xml

1 <configuration>
2 <property>
3 <name>yarn.acl.enable</name>
 4 <value>0</value>
5 </property>
6 <property>
7 <name>yarn.resourcemanager.hostname</name>
8 <value>node-master</value>
9 </property>
10 <property>
11 <name>yarn.nodemanager.aux-services</name>
12 <value>mapreduce_shuffle</value>
13 </property>
14 </configuration>

9. Duplicate the config files to each node.

10. Format the HDFS (same as pseudo-distributed).
11. Start the HDFS (same as pseudo-distributed).
12. Run YARN (same as pseudo-distributed).
Findings and Learnings:

1. We have installed HADOOP in both pseudo-distributed and fully-distributed modes

Experiment 3: Week 4:
3.Implement the following file management tasks in Hadoop:
1 Adding files and directories
2 Retrieving files
3 Deleting files
Hint: A typical Hadoop workflow creates data files (such as log files) elsewhere and copies them
into
HDFS using one of the above command line utilities.
Open a terminal window to the current working directory.
# 1. Print the Hadoop version
hadoop version
***************b) # 2. List the contents of the root directory in HDFS
hadoop fs -ls /
# 3. Report the amount of space used and
# available on currently mounted filesystem
hadoop fs -df hdfs:/
# 4. Count the number of directories,files and bytes under
# the paths that match the specified file pattern
hadoop fs -count hdfs:/
# 5. Run a DFS filesystem checking utility
hadoop fsck – /
# 6. Run a cluster balancing utility
hadoop balancer
a) # 7. Create a new directory named “hadoop” below the # /user/training directory in HDFS.
Since you’re
# currently logged in with the “training” user ID, # /user/training is your home directory in HDFS.
hadoop fs -mkdir /user/training/hadoop
# 8. Add a sample text file from the local directory
# named “data” to the new directory you created in HDFS
# during the previous step. #
hadoop fs -put data/sample.txt /user/training/hadoop
# 9. List the contents of this new directory in HDFS.
hadoop fs -ls /user/training/Hadoop
# 10. Add the entire local directory called “retail” to the
# /user/training directory in HDFS.
hadoop fs -put data/retail /user/training/hadoop
# 11. Since /user/training is your home directory in HDFS, # any command that does not have an
absolute path is
# interpreted as relative to that directory. The next
# command will therefore list your home directory, and # should show the items you’ve just added
there. hadoop fs -ls
# 12. See how much space this directory occupies in HDFS.
hadoop fs -du -s -h hadoop/retail
b) ******************# 13. Delete a file ‘customers’ from the “retail” directory.
hadoop fs -rm hadoop/retail/customers # 14. Ensure this file is no longer in HDFS. hadoop fs
-ls hadoop/retail/customers
# 15. Delete all files from the “retail” directory using a wildcard.
hadoop fs -rm hadoop/retail/*
# 16. To empty the trash
hadoop fs -expunge
# 17. Finally, remove the entire retail directory and all # of its contents in HDFS.
hadoop fs -rm -r hadoop/retail
# 18. List the hadoop directory again
hadoop fs -ls hadoop
# 19. Add the purchases.txt file from the local directory
# named “/home/training/” to the hadoop directory you created in HDFS hadoop fs -
copyFromLocal /home/training/purchases.txt hadoop/ # 20. To view the contents of your
text file purchases.txt
# which is present in your hadoop directory.
hadoop fs -cat hadoop/purchases.txt
# 21. Add the purchases.txt file from “hadoop” directory which is present in HDFS directory
# to the directory “data” which is present in your local directory
hadoop fs -copyToLocal hadoop/purchases.txt /home/training/data
# 22. cp is used to copy files between directories present in HDFS
hadoop fs -cp /user/training/*.txt /user/training/hadoop
# 23. ‘-get’ command can be used alternaively to ‘-copyToLocal’ command
hadoop fs -get hadoop/sample.txt /home/training/
# 24. Display last kilobyte of the file “purchases.txt” to stdout
hadoop fs -tail hadoop/purchases.txt
# 25. Default file permissions are 666 in HDFS
# Use ‘-chmod’ command to change permissions of a file
hadoop fs -ls hadoop/purchases.txt
sudo -u hdfs hadoop fs -chmod 600 hadoop/purchases.txt
# 26. Default names of owner and group are training,training
# Use ‘-chown’ to change owner name and group name simultaneously
hadoop fs -ls hadoop/purchases.txt
sudo -u hdfs hadoop fs -chown root:root hadoop/purchases.txt
# 27. Default name of group is training
# Use ‘-chgrp’ command to change group name
hadoop fs -ls hadoop/purchases.txt
sudo -u hdfs hadoop fs -chgrp training hadoop/purchases.txt
# 28. Move a directory from one location to other
hadoop fs -mv hadoop apache_hadoop
# 29. Default replication factor to a file is 3.
# Use ‘-setrep’ command to change replication factor of a file
hadoop fs -setrep -w 2 apache_hadoop/sample.txt
# 30. Copy a directory from one node in the cluster to another
# Use ‘-distcp’ command to copy,
# -overwrite option to overwrite in an existing files # -update command to synchronize both
directories
hadoop fs -distcp hdfs://namenodeA/apache_hadoop
hdfs://namenodeB/hadoop
# 31. Command to make the name node leave safe mode
hadoop fs -expunge
sudo -u hdfs hdfs dfsadmin -safemode leave
# 32. List all the hadoop file system shell commands
hadoop fs
# 33. Last but not least, always ask for help!
hadoop fs -help