TransWikia.com

Hadoop on VMWare - Mapreduce example program fails to run

Super User Asked by bharathyes on November 20, 2021

I am trying to run hadoop 3.2.0 on Ubuntu 18.10 running as a VM over Win10.
Want to execute a sample word count program to verify that the installation was successful and that hadoop has been properly configured.

When I try to run the included example jar by using command:

bharath@ubuntu:/usr/share/hadoop$ bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar grep input
output ‘dfs[a-z.]+’

I get this on the terminal:

2019-05-05 11:51:55,667 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2019-05-05 11:51:55,873 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2019-05-05 11:51:55,873 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2019-05-05 11:51:56,020 INFO mapreduce.JobSubmitter: Cleaning up the staging area file:/tmp/hadoop/mapred/staging/bharath705403707/.staging/job_local705403707_0001
ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:234)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:861)
    at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:508)
    at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:489)
    at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:511)
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:676)
    at org.apache.hadoop.mapreduce.JobResourceUploader.mkdirs(JobResourceUploader.java:658)
    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:172)
    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:133)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
    at org.apache.hadoop.examples.Grep.run(Grep.java:78)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.examples.Grep.main(Grep.java:103)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

I am new to hadoop and even Java so I am unable to make much sense of what any of this means other than the fact that the program isn’t working and that certain files might be missing.

Can anyone point out what the issue is here or make suggestions to find what is causing this issue?

2 Answers

The key is that the input file location is on hdfs not on your local hard (on your computer). Thus, the address you entered is wrong and you should: First put the input file on hdfs with its command. Second run the right command to execute that file. BR, Neda

Answered by Neda Maleki on November 20, 2021

There seems to be an error with the command. Syntax is: hadoop jar jar-file class-with-Main input-file-hdfs output-dir-hdfs
Output dir cannot pre exist

Ex:

hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvolume/in /myvolume/out

Answered by Suyash Limaye on November 20, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP