Session 03 - Hadoop Installation and Basic Commands

Hadoop Developer Training
Session 3 -
Installation and Commands

Classification: Restricted
Agenda
• Hadoop Installation and Commands

Installation Guide
• Step:- 1 Copy hadoop installation files from local to virtual machine
Command :-
hadoop@hadoop-VirtualBox:~$ sudo cp -r /media/sf_Dee/
/home/hadoop/Desktop/
• Step:- 2 Give permission to folder on Desktop
Command :-
hadoop@hadoop -VirtualBox:~$ sudo chmod 777 -R
/home/hadoop/Desktop/sf_Dee/
[sudo] password for hadoop:
• Step:- 3 Make a Folder work in /usr/local/work install hadoop and jave
inside it .
Command:-
hadoop@hadoop-VirtualBox:~$ sudo mkdir /usr/local/work

Installation Guide
• Step:- 4 Copy the hadoop and java.tar files to the work folder.
Command:-
hadoop@hadoop-VirtualBox:~$ sudo cp -r
/home/hadoop/Desktop/sf_Dee/jdk-8u60-linux-x64.tar.gz /usr/local/work/
hadoop@hadoop-VirtualBox:~$ sudo cp -r
/home/hadoop/Desktop/sf_Dee/de/Setups/hadoop-2.6.0.tar.gz
/usr/local/work/
hadoop@hadoop-VirtualBox:~$ cd /usr/local/work/
• Step:- 5 Untar the java & hadoop.tar files inside work folder
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work$ sudo tar -xzvf jdk-8u60-linux-
x64.tar.gz
hadoop@hadoop-VirtualBox:/usr/local/work$ sudo tar -xzvf hadoop-
2.6.0.tar.gz

Installation Guide
• Step:- 6 Move the hadoop-2.6.0 and jdk1.8.0_60 folder as hadoop and java.
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work$ sudo mv hadoop-2.6.0
/usr/local/work/hadoop
hadoop@hadoop-VirtualBox:/usr/local/work$ sudo mv jdk1.8.0_60
/usr/local/work/java
• Step:- 7 Install rsync & ssh
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work$ sudo apt-get install ssh
hadoop@hadoop-VirtualBox:/usr/local/work$ sudo apt-get install rsync

Installation Guide
• Step:- 8 Generate and copy keys to make passwordless connection
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work$ ssh-keygen -t dsa -P '' -f
~/.ssh/id_dsa
hadoop@hadoop-VirtualBox:/usr/local/work$ cat ~/.ssh/id_dsa.pub >>
~/.ssh/authorized_keys
• Step:- 9 Update java to os
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work$ sudo update-alternatives --
install "/usr/bin/java" "java" "/usr/local/work/java/bin/java" 1 update-
alternatives: using /usr/local/work/java/bin/java to provide /usr/bin/java
(java) in auto mode

Installation Guide
install "/usr/bin/javac" "javac" "/usr/local/work/java/bin/javac" 1 update-
alternatives: using /usr/local/work/java/bin/javac to provide /usr/bin/javac
(javac) in auto mode
install "/usr/bin/javaws" "javaws" "/usr/local/work/java/bin/javaws" 1
update-alternatives: using /usr/local/work/java/bin/javaws to provide
/usr/bin/javaws (javaws) in auto mode
• Step:- 10 Set java to OS
Command:-

Installation Guide
• Step:- 11 Update java & hadoop path in bashrc profile (env variables at the
end of doc)
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work$ sudo nano ~/.bashrc
hadoop@hadoop-VirtualBox:/usr/local/work$ source ~/.bashrc
• Step:- 12 Check if the hadoop and java are installed perfectly.
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work$ echo $HADOOP_PREFIX
output would be :- /usr/local/work/hadoop/

Installation Guide
hadoop@hadoop-VirtualBox:/usr/local/work$ echo $JAVA_HOME
output would be :- /usr/local/work/java
• Step:- 13 Copy the mapred-site.xml.template to mapred-site.xml in
/usr/local/work/hadoop/etc/hadoop
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work$
hadoop@hadoop-VirtualBox:/usr/local/work/hadoop/etc/hadoop$ sudo cp
mapred-site.xml.template mapred-site.xml

Installation Guide
• Step:- 14 Edit hadoop-env.sh file (you should be inside the directory
/usr/local/work/hadoop/etc/hadoop)
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work/hadoop/etc/hadoop$ sudo
nano hadoop-env.sh
enter/change java path to your path of java installation :
export JAVA_HOME=/usr/local/work/java
• Step:- 15 Edit core-site.xml file (you should be inside the directory
Command:-
nano core-site.xml

Installation Guide
Enter the following properties inside it
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:8020</value>
<final>true</final>
</property>
</configuration>
• Step:- 16 Edit hdfs-site.xml file (you should be inside the directory
Command:-
nano hdfs-site.xml

Installation Guide
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/work/hadoop/hadoop_data/dfs/name</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>268435456</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/work/hadoop/hadoop_data/dfs/data</value>
</property>
</configuration>

Installation Guide
• Step:- 17 Edit mapred-site.xml file (you should be inside the directory
/usr/local/work/hadoop/etc/hadoop) Command:-
nano mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

Installation Guide
• Step:- 18 Edit yarn-mapred.xml file (you should be inside the directory
/usr/local/work/hadoop/etc/hadoop) Command:-
nano yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.address</name>
<value>localhost:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
</property>

Installation Guide
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.Capacity
Scheduler</value>
</property>

Installation Guide
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.Capacity
Scheduler</value>
</property>
<property> <name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value> </property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>file:/usr/local/work/hadoop/hadoop_data/yarn/yarn.nodemanager.local-
dirs</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>file:/usr/local/work/hadoop/hadoop_data/yarn/logs</value>
</property>

Installation Guide
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
• Step:- 19 Format name node
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work/hadoop/etc/hadoop$ hadoop
namenode -format
• Step:- 20 Start services of hadoop
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work/hadoop/etc/hadoop$ start-
all.sh

Installation Guide
Following services would start
• Namenode
• Secondarynamenode
• Datanode
• Resourcemanger
• Nodemanger
• Jps
• Step:- 21 Stop all services
Command:-
hadoop@hadoop-VirtualBox:/usr/local/work/hadoop/etc/hadoop$ stop-
all.sh

Installation Guide
Note :- variable to be added in ~/.bashrc profile
• export HADOOP_PREFIX="/usr/local/work/hadoop/"
• export PATH=$PATH:$HADOOP_PREFIX/bin
• export PATH=$PATH:$HADOOP_PREFIX/sbin
• export HADOOP_COMMON_HOME=${HADOOP_PREFIX}
• export HADOOP_MAPRED_HOME=${HADOOP_PREFIX}
• export HADOOP_HDFS_HOME=${HADOOP_PREFIX}
• export YARN_HOME=${HADOOP_PREFIX}
• export JAVA_HOME="/usr/local/work/java"
• export PATH=$PATH:$JAVA_HOME/bin

Topics to be discussed in the next session
• PIG
• PIG - Overview
• Installation and Running Pig
• Load in Pig
• Macros in Pig

Thank you!

Session 03 - Hadoop Installation and Basic Commands

Recommended

More Related Content

What's hot (20)

Similar to Session 03 - Hadoop Installation and Basic Commands (20)

More from AnandMHadoop (8)

Recently uploaded (20)

Session 03 - Hadoop Installation and Basic Commands