0% found this document useful (0 votes)
25 views

Exp2 Hadoop

The document shows the steps taken to run a word count MapReduce job on a Hadoop cluster. It includes starting HDFS and YARN, placing an input file in HDFS, running the wordcount example job, and viewing the output. The job counts the number of occurrences of each word in the input file and writes the results to the output directory in HDFS.

Uploaded by

Abdul Wajeed
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
25 views

Exp2 Hadoop

The document shows the steps taken to run a word count MapReduce job on a Hadoop cluster. It includes starting HDFS and YARN, placing an input file in HDFS, running the wordcount example job, and viewing the output. The job counts the number of occurrences of each word in the input file and writes the results to the output directory in HDFS.

Uploaded by

Abdul Wajeed
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 6

C:\Windows\System32>cd..

C:\Windows>cd..

C:\>cd C:\hadoop-3.2.4\bin

C:\hadoop-3.2.4\bin>cd..

C:\hadoop-3.2.4>cd sbin

C:\hadoop-3.2.4\sbin>start-dfs

C:\hadoop-3.2.4\sbin>start-yarn
starting yarn daemons

C:\hadoop-3.2.4\sbin>jps
13156 Jps
3828 DataNode
5224 NameNode
8424 NodeManager
3388 ResourceManager

C:\hadoop-3.2.4\sbin>hadoop fs -mkdir /input

C:\hadoop-3.2.4\sbin>hadoop fs -put C:/hadoop-3.2.4/data.txt /input

C:\hadoop-3.2.4\sbin>hadoop fs -ls /input/


Found 1 items
-rw-r--r-- 1 ASUS supergroup 39 2024-01-30 15:43 /input/data.txt

C:\hadoop-3.2.4\sbin>hadoop dfs -cat /input/data.txt


DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Hello
Hi
Hello
Good morning
Teacher
C:\hadoop-3.2.4\sbin>hadoop jar

C:\hadoop-3.2.4\sbin>hadoop jar C:/hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-


3.2.4.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the
input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
wordmean: A map/reduce program that counts the average length of the words in the input files.
wordmedian: A map/reduce program that counts the median length of the words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in
the input files.

C:\hadoop-3.2.4\sbin>hadoop jar C:/hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-


3.2.4.jar wordcount /input /out
2024-01-30 15:53:02,627 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2024-01-30 15:53:03,652 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-
yarn/staging/ASUS/.staging/job_1706608858316_0001
2024-01-30 15:53:03,937 INFO input.FileInputFormat: Total input files to process : 1
2024-01-30 15:53:04,042 INFO mapreduce.JobSubmitter: number of splits:1
2024-01-30 15:53:04,229 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1706608858316_0001
2024-01-30 15:53:04,231 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-01-30 15:53:04,462 INFO conf.Configuration: resource-types.xml not found
2024-01-30 15:53:04,462 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-01-30 15:53:05,138 INFO impl.YarnClientImpl: Submitted application application_1706608858316_0001
2024-01-30 15:53:05,187 INFO mapreduce.Job: The url to track the job: http://DESKTOP-
1969SCE:8088/proxy/application_1706608858316_0001/
2024-01-30 15:53:05,188 INFO mapreduce.Job: Running job: job_1706608858316_0001
2024-01-30 15:53:19,741 INFO mapreduce.Job: Job job_1706608858316_0001 running in uber mode : false
2024-01-30 15:53:19,755 INFO mapreduce.Job: map 0% reduce 0%
2024-01-30 15:53:27,941 INFO mapreduce.Job: map 100% reduce 0%
2024-01-30 15:53:36,070 INFO mapreduce.Job: map 100% reduce 100%
2024-01-30 15:53:36,079 INFO mapreduce.Job: Job job_1706608858316_0001 completed successfully
2024-01-30 15:53:36,225 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=66
FILE: Number of bytes written=478269
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=140
HDFS: Number of bytes written=40
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5401
Total time spent by all reduces in occupied slots (ms)=5719
Total time spent by all map tasks (ms)=5401
Total time spent by all reduce tasks (ms)=5719
Total vcore-milliseconds taken by all map tasks=5401
Total vcore-milliseconds taken by all reduce tasks=5719
Total megabyte-milliseconds taken by all map tasks=5530624
Total megabyte-milliseconds taken by all reduce tasks=5856256
Map-Reduce Framework
Map input records=5
Map output records=6
Map output bytes=60
Map output materialized bytes=66
Input split bytes=101
Combine input records=6
Combine output records=5
Reduce input groups=5
Reduce shuffle bytes=66
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=142
CPU time spent (ms)=809
Physical memory (bytes) snapshot=465309696
Virtual memory (bytes) snapshot=646627328
Total committed heap usage (bytes)=317194240
Peak Map Physical memory (bytes)=287711232
Peak Map Virtual memory (bytes)=400330752
Peak Reduce Physical memory (bytes)=185565184
Peak Reduce Virtual memory (bytes)=252416000
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=39
File Output Format Counters
Bytes Written=40

C:\hadoop-3.2.4\sbin>hadoop fs -cat /out/*


Good 1
Hello 2
Hi 1
Teacher 1
morning 1

C:\hadoop-3.2.4\sbin>

You might also like