Big Data Hadoop Local and Public Cloud (Amazon EMR)

Danairat T., 2013, danairat@gmail.comBig Data Hadoop – Hands On Workshop 1
Big Data Hadoop
Local and Public Cloud
Hands On Workshop
Dr.Thanachart Numnonda
thanachart@imcinstitute.com
Danairat T.
Certified Java Programmer, TOGAF – Silver
danairat@gmail.com, +66-81-559-1446

Thanachart Numnonda and Danairat T, July 2013Big Data Hadoop – Hands On Workshop 2
Hands-On: Running Hadoop
on Amazon Elastic MapReduce

Architecture Overview of Amazon EMR

Creating an AWS account

Signing up for the necessary services
●
Simple Storage Service (S3)
●
Elastic Compute Cloud (EC2)
●
Elastic MapReduce (EMR)
Caution! This costs real money!

Creating Amazon EC2 Instance

Creating Amazon S3 bucket

Create access key using Security Credentials
in the AWS Management Console

Creating a new Job Flow in EMR

View Result from the S3 bucket

Lecture: Understanding Map Reduce
Processing
Client
Name Node Job Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Map Reduce

MapReduce Framework
map: (K1, V1) -> list(K2, V2))
reduce: (K2, list(V2)) -> list(K3, V3)

MapReduce Processing – The Data
flow
1. InputFormat, InputSplits, RecordReader
2. Mapper - your focus is here
3. Partition, Shuffle & Sort
4. Reducer - your focus is here
5. OutputFormat, RecordWriter

How does the MapReduce work?
Output in a list of (Key, List of Values)
in the intermediate file
Sorting
Partitioning
Output in a list of (Key, Value)
InputSplit
RecordReader
RecordWriter

How does the MapReduce work?
Sorting
Partitioning
Combining
Car, 2
Car, 2
Bear, {1,1}
Car, {2,1}
River, {1,1}
Deer, {1,1}
Output in a list of (Key, List of Values)
Output in a list of (Key, Value)
InputSplit
RecordReader
RecordWriter

Hands-On: Writing you own Map
Reduce Program

Wordcount (HelloWord in Hadoop)
1. package org.myorg;
2.
3. import java.io.IOException;
4. import java.util.*;
5.
6. import org.apache.hadoop.fs.Path;
7. import org.apache.hadoop.conf.*;
8. import org.apache.hadoop.io.*;
9. import org.apache.hadoop.mapred.*;
10. import org.apache.hadoop.util.*;
11.
12. public class WordCount {
13.
14.
public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text,
IntWritable> {
15. private final static IntWritable one = new IntWritable(1);
16. private Text word = new Text();
17.
18.
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter
reporter) throws IOException {
19. String line = value.toString();
20. StringTokenizer tokenizer = new StringTokenizer(line);
21. while (tokenizer.hasMoreTokens()) {
22. word.set(tokenizer.nextToken());
23. output.collect(word, one);
24. }
25. }
26. }

27.
28. public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text,
IntWritable> {
29.
public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable>
output, Reporter reporter) throws IOException {
30. int sum = 0;
31. while (values.hasNext()) {
32. sum += values.next().get();
33. }
34. output.collect(key, new IntWritable(sum));
35. }
36. }
37.

38. public static void main(String[] args) throws Exception {
39. JobConf conf = new JobConf(WordCount.class);
40. conf.setJobName("wordcount");
41.
42. conf.setOutputKeyClass(Text.class);
43. conf.setOutputValueClass(IntWritable.class);
44.
45. conf.setMapperClass(Map.class);
46.
47. conf.setReducerClass(Reduce.class);
48.
49. conf.setInputFormat(TextInputFormat.class);
50. conf.setOutputFormat(TextOutputFormat.class);
51.
52. FileInputFormat.setInputPaths(conf, new Path(args[0]));
53. FileOutputFormat.setOutputPath(conf, new Path(args[1]));
54.
55. JobClient.runJob(conf);
57. }
58. }
59.

Hands-On: Packaging Map Reduce
and Deploying to Hadoop Runtime
Environment

Packaging Map Reduce Program
Usage
Assuming HADOOP_HOME is the root of the installation and HADOOP_VERSION is the Hadoop version
installed, compile WordCount.java and create a jar:
$ mkdir /home/hduser/wordcount_classes
$ cd /home/hduser
$ javac -classpath /usr/local/hadoop/hadoop-core-0.20.205.0.jar -d wordcount_classes WordCount.java
$ jar -cvf ./wordcount.jar -C wordcount_classes/ .
$ hadoop jar ./wordcount.jar org.myorg.WordCount /input/* /output/wordcount_output_dir
Output:
…….
$ hadoop dfs -cat /output/wordcount_output_dir/part-00000

Hands-On: Running WordCount.jar
on Amazon EMR

Upload .jar file and input file to
Amazon S3
1. Select <yourbucket> in Amazon S3 service
2. Create folder : applications
3. Upload wordcount.jar to the applications folder
4. Create another folder: input
5. Upload input_test.txt to the input folder

Running a new Job Flow in EMR

Input JAR Location and Arguments

View the Result

Hands-On: Analytics Using
MapReduce

Three Analytic MapReduce Examples
1. Simple analytics using MapReduce
2. Performing Group-By using MapReduce
3. Calculating frequency distributions and
sorting using MapReduce

NASA weblog dataset available from
http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html
is a real-life dataset collected using the requests received by NASA web servers.
Download the weblog dataset from ftp://ita.ee.lbl.gov/traces/NASA_
access_log_Jul95.gz and unzip it. We call the extracted folder as DATA_DIR.
$ hadoopdfs -mkdir /data
$ hadoopdfs -put <DATA_DIR>/NASA_access_log_Jul95 /data/input1
Preparing Example Data

Aggregative values (for example, Mean, Max, Min, standard deviation, and so on)
provide the basic analytics about a dataset..
Simple analytics using MapReduce
Source: Hadoop MapReduce CookBook

package analysis;
import java.io.IOException;
import java.util.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
import org.apache.hadoop.conf.*;
public class WebLogMessageSizeAggregator {
public static final Pattern httplogPattern = Pattern
.compile("([^s]+) - - [(.+)] "([^s]+) (/[^s]*) HTTP/[^s]+"
[^s]+ ([0-9]+)");
public static class AMapper extends MapReduceBase implements Mapper<LongWritable,
Text, Text, IntWritable> {
WebLogMessageSizeAggregator.java

public void map(LongWritable key, Text value, OutputCollector<Text,
IntWritable> output,Reporter reporter) throws IOException {
Matcher matcher = httplogPattern.matcher(value.toString());
if (matcher.matches()) {
int size = Integer.parseInt(matcher.group(5));
output.collect(new Text("msgSize"), new IntWritable(size));
}
}
}

public static class AReducer extends MapReduceBase implements Reducer<Text,
IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,Reporter reporter) throws IOException {
double tot = 0;
int count = 0;
int min = Integer.MAX_VALUE;
int max = 0;
while (values.hasNext()) {
int value = values.next().get();
tot = tot + value;
count++;
if (value < min) {
min = value;
}
if (value > max) {
max = value;
}
}

output.collect(new Text("Mean"), new IntWritable((int) tot / count));
output.collect(new Text("Max"), new IntWritable(max));
output.collect(new Text("Min"), new IntWritable(min));
}
}
public static void main(String[] args) throws Exception {
JobConf job = new JobConf(WebLogMessageSizeAggregator.class);
job.setJarByClass(WebLogMessageSizeAggregator.class);
job.setMapperClass(AMapper.class);
job.setReducerClass(AReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
JobClient.runJob(job);
}
}

Compile, Build JAR, Submit Job, Review Result
$ cd /home/hduser
$ javac -classpath /usr/local/hadoop/hadoop-core-0.20.205.0.jar -d WebLog WebLogMessageSizeAggregator.java
$ jar -cvf ./weblog.jar -C WebLog .
$ hadoop jar ./weblog.jar analysis.WebLogMessageSizeAggregator /data/* /output/result_weblog
Output:
......
$ hadoop dfs -cat /output/result_weblog/part-00000

A MapReduce to group data into simple groups and calculate the analytics for
each group.
Performing Group-By using MapReduce

public class WeblogHitsByLinkProcessor {
public static final Pattern httplogPattern = Pattern
.compile("([^s]+) - - [(.+)] "([^s]+) (/[^s]*) HTTP/[^s]+"
[^s]+ ([0-9]+)");
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
Matcher matcher = httplogPattern.matcher(value.toString());
if (matcher.matches()) {
String linkUrl = matcher.group(4);
word.set(linkUrl);
output.collect(word, one);
}
}
}
WeblogHitsByLinkProcessor.java

private IntWritable result = new IntWritable();
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
result.set(sum);
output.collect(key, result);
}
}
WeblogHitsByLinkProcessor.java

$ cd /home/hduser
$ javac -classpath /usr/local/hadoop/hadoop-core-0.20.205.0.jar -d WebLogHit WeblogHitsByLinkProcessor.java
$ jar -cvf ./webloghit.jar -C WebLogHit .
$ hadoop jar ./webloghit.jar analysis.WeblogHitsByLinkProcessor /data/* /output/result_webloghit
Output:
......
$ hadoop dfs -cat /output/result_webloghit/part-00000

Frequency distribution is the number of hits received by each URL sorted in the
Ascending order, by the number hits received by a URL. We have already calculated
the number of hits inthe previous program.
Calculating frequency distributions and
sorting using MapReduce

public class WeblogFrequencyDistributionProcessor {
public static final Pattern httplogPattern = Pattern.compile("([^s]+) - - [(.
+)] "([^s]+) (/[^s]*) HTTP/[^s]+" [^s]+ ([0-9]+)");
String[] tokens = value.toString().split("s");
output.collect(new Text(tokens[0]),new
IntWritable(Integer.parseInt(tokens[1])));
}
}
WeblogFrequencyDistributionProcessor.java

/**
* <p>Reduce function receives all the values that has the same key as the input,
and it output the key
* and the number of occurrences of the key as the output.</p>
*/
if(values.hasNext()){
output.collect(key, values.next());
}
}
}
WeblogFrequencyDistributionProcessor.java

$ cd /home/hduser
$ javac -classpath /usr/local/hadoop/hadoop-core-0.20.205.0.jar -d WebLogFreq
WWeblogFrequencyDistributionProcessor.java
$ jar -cvf ./weblogfreq.jar -C WebLogFreq .
$ hadoop jar ./weblogfreq.jar analysis.WeblogFrequencyDistributionProcessor /output/result_webloghit/*
/output/result_weblogfreq
Output:
......
$ hadoop dfs -cat /output/result_weblogfreq/part-00000

Exercise: Runming the analytic
programs on Amazon EMR.

Thank you

Big Data Hadoop Local and Public Cloud (Amazon EMR)

More Related Content

What's hot (20)

Similar to Big Data Hadoop Local and Public Cloud (Amazon EMR) (20)

More from IMC Institute (20)

Recently uploaded (20)

Big Data Hadoop Local and Public Cloud (Amazon EMR)