Hadoop Workshop using Cloudera on Amazon EC2

Danairat T., 2013, danairat@gmail.comBig Data Hadoop – Hands On Workshop 1
Hadoop Workshop using
Cloudera on Amazon EC2
May 2015
Dr.Thanachart Numnonda
IMC Institute
thanachart@imcinstitute.com
Modifiy from Original Version by Danairat T.
Certified Java Programmer, TOGAF – Silver
danairat@gmail.com

Danairat T., , danairat@gmail.com: Thanachart Numnonda, thanachart@imcinstitute.com May 2015Hadoop Workshop using Cloudera on Amazon EC2
Hands-On: Launch a virtual server
on EC2 Amazon Web Services

Virtual Server
This lab will use a EC2 virtual server to install a
Hadoop server using the following features:
1. Ubuntu Server 14.04 LTS
2. m3.xLarge 4vCPU, 15 GB memory, 80 GB SSD
3. Security group: create new
4. Keypair: imchadoop

Select a EC2 service and click on Lunch Instance

Select an Amazon Machine Image (AMI) and
Ubuntu Server 14.04 LTS (PV)

Choose m3.xlarge Type virtual server

Leave configuration details as default

Add Storage: 30 GB

Name the instance

Select Create a new security group > Add Rule as
follows

Click Launch and choose imchadoop as a key pair

Review an instance / click Connect for
an instruction to connect to the instance

Connect to an instance from Mac/Linux

Connect to an instance from Windows using Putty

Connect to the instance

Hands-On: Installing Cloudera on EC2

Download Cloudera Manager
1) Type command >wget
http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin
2) Type command > chmod u+x cloudera-manager-installer.bin
3) Type command > sudo ./cloudera-manager-installer.bin

Login to Cloudera Manager
Wait several minutes for the Cloudera Manager Server to complete its startup.
Then running web browser: http:// public-ip: 7180

Select Cloudera Express Edition

Provide your instance <public ip>
addresses in the cluster

Browse the private key (imchadoop.pem) file which we have downloaded in the
previous part. Keep Passphrase as blank

If you see the above error, DO NOT worry at all, it’s known issue. You can find
the known issue list at Cloudera Issue List.
Click “Back” button until home screen then click “Continue” button

Now you will find a tab “Currently Managed Hosts” with their private dns and
private ip address. Select all and click “Continue”

Finish

Running Hue

Sign in to Hue

Starting Hue on Cloudera

Viewing HDFS

Hands-On: Importing/Exporting
Data to HDFS

Importing Data to Hadoop
Download War and Peace Full Text
www.gutenberg.org/ebooks/2600
$hadoop fs -mkdir input
$hadoop fs -mkdir output
$hadoop fs -copyFromLocal Downloads/pg2600.txt input

Review file in Hadoop HDFS
[hdadmin@localhost bin]$ hadoop fs -cat input/pg2600.txt
List HDFS File
Read HDFS File
Retrieve HDFS File to Local File System
Please see also http://hadoop.apache.org/docs/r1.0.4/commands_manual.html
[hdadmin@localhost bin]$ hadoop fs -copyToLocal input/pg2600.txt tmp/file.txt

Review file in Hadoop HDFS using
File Browse

Review file in Hadoop HDFS using Hue

Hadoop Port Numbers
Daemon Default
Port
Configuration Parameter in
conf/*-site.xml
HDFS Namenode 50070 dfs.http.address
Datanodes 50075 dfs.datanode.http.address
Secondarynamenode 50090 dfs.secondary.http.address
MR JobTracker 50030 mapred.job.tracker.http.addre
ss
Tasktrackers 50060 mapred.task.tracker.http.addr
ess

Removing data from HDFS using
Shell Command
hdadmin@localhost detach]$ hadoop fs -rm input/pg2600.txt
Deleted hdfs://localhost:54310/input/pg2600.txt
hdadmin@localhost detach]$

Hands-On: Writing Map/Reduce
Program on Eclipse

Starting Eclipse in Cloudera VM

Create a Java Project
Let's name it HadoopWordCount

Add dependencies to the project
●
Add the following two JARs to your build path
●
hadoop-common.jar and hadoop-mapreduce-client-core.jar. Both can be
founded at /usr/lib/hadoop/client
●
By perform the following steps
– Add a folder named lib to the project
– Copy the mentioned JARs in this folder
– Right-click on the project name >> select Build Path >> then
Configure Build Path
– Click on Add Jars, select these two JARs from the lib folder

Add dependencies to the project

Writing a source code
●
Right click the project, the select New >> Package
●
Name the package as org.myorg
●
Right click at org.myorg, the select New >> Class
●
Name the package as WordCount
●
Writing a source code as shown in previoud slides

Building a Jar file
●
Right click the project, the select Export
●
Select Java and then JAR file
●
Provide the JAR name, as wordcount.jar
●
Leave the JAR package options as default
●
In the JAR Manifest Specification section, in the botton, specify the Main
class
●
In this case, select WordCount
●
Click on Finish
●
The JAR file will be build and will be located at cloudera/workspace
Note: you may need to re-size the dialog font size by select
Windows >> Preferences >> Appearance >> Colors and Fonts

Hands-On: Running Map Reduce and
Deploying to Hadoop Runtime
Environment

Running Map Reduce Program

Reviewing MapReduce Job in Hue

Reviewing MapReduce Output Result

Hands-On: Running Map Reduce
using Oozie workflow

Using Hue: select WorkFlow >> Editor

Create a new workflow
●
Click Create button; the following screen will be displayed
●
Name the workflow as WordCountWorkflow

Select a Java job for the workflow
●
From the Oozie editor, drag Java and drop between start and end

Edit the Java Job
●
Assign the following value
– Name: WordCount
– Jar name: wordcount.jar (select … choose upload from local machine)
– Main Class: org.myorg.WordCount
– Arguments: input/* output/wordcount_output2

Submit the workflow
●
Click Done, follow by Save
●
Then click submit

Hands-On: Working with a csv data

A sample CSV data
●
The input data is access logs with the following form
Date, Requesting-IP-Address
●
We will write a map reduce program to count the number of hits to the
website per country.

HitsByCountryMapper.java
package learning.bigdata.mapreduce;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class HitsByCountryMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static String[] COUNTRIES = { "India", "UK", "US", "China" };
private Text outputKey = new Text();
private IntWritable outputValue = new IntWritable();
@Override
protected void setup(Context context) throws IOException, InterruptedException {
super.setup(context);
}
@Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
try {
String valueString = value.toString();
// Split the value string to get Date and ipAddress
String[] row = valueString.split(",");

HitsByCountryMapper.java
// row[0]= Date and row[1]=ipAddress
String ipAddress = row[1];
// Get the country name to which the ipAddress belongs
String countryName = getCountryNameFromIpAddress(ipAddress);
outputKey.set(countryName);
outputValue.set(1);
context.write(outputKey, outputValue);
} catch (ArrayIndexOutOfBoundsException ex) {
context.getCounter("Custom counters", "MAPPER_EXCEPTION_COUNTER").increment(1);
ex.printStackTrace();
}
}
private static String getCountryNameFromIpAddress(String ipAddress) {
if (ipAddress != null && !ipAddress.isEmpty()) {
int randomIndex = Math.abs(ipAddress.hashCode()) % COUNTRIES.length;
return COUNTRIES[randomIndex];
}
return null;
}
}

HitsByCountryReducer.java
package learning.bigdata.mapreduce;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.mapreduce.Reducer;
public class HitsByCountryReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
private Text outputKey = new Text();
private IntWritable outputValue = new IntWritable();
private int count = 0;
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException,
InterruptedException {
count = 0;
Iterator<IntWritable> iterator = values.iterator();
while (iterator.hasNext()) {
IntWritable value = iterator.next();
count += value.get();
}
outputKey.set(key);
outputValue.set(count);
context.write(outputKey, outputValue);
}
}

HitsByCountry.java
package learning.bigdata.main;
import learning.bigdata.mapreduce.HitsByCountryMapper;
import learning.bigdata.mapreduce.HitsByCountryReducer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class HitsByCountry extends Configured implements Tool {
private static final String JOB_NAME = "Calculating hits by country";
public static void main(String[] args) throws Exception {
if (args.length < 2) {
System.out.println("Usage: HitsByCountry <comma separated input directories> <output dir>");
System.exit(-1);
}
int result = ToolRunner.run(new HitsByCountry(), args);
System.exit(result);
}

HitsByCountry.java
@Override
public int run(String[] args) throws Exception {
try {
Configuration conf = getConf();
Job job = Job.getInstance(conf);
job.setJarByClass(HitsByCountry.class);
job.setJobName(JOB_NAME);
job.setMapperClass(HitsByCountryMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setReducerClass(HitsByCountryReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.setInputPaths(job, args[0]);
FileOutputFormat.setOutputPath(job, new Path(args[1]));
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
} catch (Exception e) {
e.printStackTrace();
return 1;
}
}
}

Lecture: Developing Complex
Hadoop MapReduce
Applications

Choosing appropriate Hadoop data types
●
Hadoop uses the Writable interface based classes as
the data types for the MapReduce computations.
●
Choosing the appropriate Writable data types for your
input, intermediate, and output data can have a large
effect on the performance and the programmability of
your MapReduce programs.
●
In order to be used as a value data type, a data type
must implement the org.apache.hadoop.io.Writable
interface.
●
In order to be used as a key data type, a data type must
implement the
org.apache.hadoop.io.WritableComparable<T> interface

Examples

Hadoop built-in data types
●
Text: This stores a UTF8 text
●
BytesWritable: This stores a sequence of bytes
●
VIntWritable and VLongWritable: These store variable
length integer and long values
●
NullWritable: This is a zero-length Writable type that can
be used when you don't want to use a key or value type

Hadoop built-in data types
●
The following Hadoop build-in collection data types can
only be used as value types.
– ArrayWritable: This stores an array of values belonging to a
Writable type.
– TwoDArrayWritable: This stores a matrix of values belonging to
the same Writable type.
– MapWritable: This stores a map of key-value pairs. Keys and
values should be of the Writable data types.
– SortedMapWritable: This stores a sorted map of key-value
pairs. Keys should implement the WritableComparable
interface.

Implementing a custom Hadoop Writable
data type
●
we can easily write a custom Writable data type by
implementing the org.apache.hadoop.io.Writable
interface
●
The Writable interface-based types can be used as
value types in Hadoop MapReduce computations.

Examples

Choosing a suitable Hadoop InputFormat
for your input data format
●
Hadoop supports processing of many different formats
and types of data through InputFormat.
●
The InputFormat of a Hadoop MapReduce computation
generates the key-value pair inputs for the mappers by
parsing the input data.
●
InputFormat also performs the splitting of the input data
into logical partitions

InputFormat that Hadoop provide
●
TextInputFormat: This is used for plain text files.
TextInputFormat generates a key-value record for each
line of the input text files.
●
NLineInputFormat: This is used for plain text files.
NlineInputFormat splits the input files into logical splits
of fixed number of lines.
●
SequenceFileInputFormat: For Hadoop Sequence file
input data
●
DBInputFormat: This supports reading the input data for
MapReduce computation from a SQL table.

Implementing new input data formats
●
Hadoop enables us to implement and specify custom
InputFormat implementations for our MapReduce
computations.
●
A InputFormat implementation should extend the
org.apache.hadoop.mapreduce.InputFormat<K,V>
abstract class
●
overriding the createRecordReader() and getSplits()
methods.

Formatting the results of MapReduce
computations – using Hadoop
OutputFormats
●
it is important to store the result of a MapReduce
computation in a format that can be consumed
efficiently by the target application
●
We can use Hadoop OutputFormat interface to define
the data storage format
●
A OutputFormat prepares the output location and
provides a RecordWriter implementation to perform the
actual serialization and storage of the data.
●
Hadoop uses the
org.apache.hadoop.mapreduce.lib.output.
TextOutputFormat<K,V> as the default OutputFormat

Hands-On: Analytics Using
MapReduce

Lecture
Understanding HBase

Introduction
An open source, non-relational, distributed database
HBase is an open source, non-relational, distributed database
modeled after Google's BigTable and is written in Java. It is
developed as part of Apache Software Foundation's Apache
Hadoop project and runs on top of HDFS (, providing
BigTable-like capabilities for Hadoop. That is, it provides a
fault-tolerant way of storing large quantities of sparse data.

HBase Features
●
Hadoop database modelled after Google's Bigtable
●
Column oriented data store, known as Hadoop Database
●
Support random realtime CRUD operations (unlike
HDFS)
●
No SQL Database
●
Opensource, written in Java
●
Run on a cluster of commodity hardware
Hive.apache.org

When to use Hbase?
●
When you need high volume data to be stored
●
Un-structured data
●
Sparse data
●
Column-oriented data
●
Versioned data (same data template, captured at various
time, time-elapse data)
●
When you need high scalability
Hive.apache.org

Which one to use?
●
HDFS
●
Only append dataset (no random write)
●
Read the whole dataset (no random read)
●
HBase
●
Need random write and/or read
●
Has thousands of operation per second on TB+ of data
●
RDBMS
●
Data fits on one big node
●
Need full transaction support
●
Need real-time query capabilities
Hive.apache.org

HBase Components
Hive.apache.org
●
Region
●
Row of table are stores
●
Region Server
●
Hosts the tables
●
Master
●
Coordinating the Region
Servers
●
ZooKeeper
●
HDFS
●
API
●
The Java Client API

HBase Shell Commands
Hive.apache.org

Hands-On: Running HBase

Starting HBase shell
[hdadmin@localhost ~]$
[hdadmin@localhost ~]$ hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.94.10, r1504995, Fri Jul 19 20:24:16 UTC 2013
hbase(main):001:0>

Create a table and insert data in HBase
hbase(main):009:0> create 'test', 'cf'
0 row(s) in 1.0830 seconds
hbase(main):010:0> put 'test', 'row1', 'cf:a', 'val1'
hbase(main):011:0> scan 'test'
ROW COLUMN+CELL
row1 column=cf:a, timestamp=1375363287644,
value=val1
hbase(main):002:0> get 'test', 'row1'
COLUMN CELL
cf:a timestamp=1375363287644, value=val1

Using Data Browsers in Hue for HBase

Recommendation to Further Study

Thank you
www.imcinstitute.com
www.facebook.com/imcinstitute

Hadoop Workshop using Cloudera on Amazon EC2

Recommended

More Related Content

What's hot (20)

Similar to Hadoop Workshop using Cloudera on Amazon EC2 (18)

More from IMC Institute (20)

Recently uploaded (20)

Hadoop Workshop using Cloudera on Amazon EC2