Class not found in Hadoop job

2015-05-26T07:55:34

I have a map reduce job which gets its input from DocumentDB. I've added to jar files under the lib directory in my source code and also user the -libjars when running the job. but I still get the class not found error for a class in the jar file. Here is some part of my driver program

public class MapReduceDriver extends Configured implements Tool  {

public static void main(String[] args) throws Exception {

    int res = ToolRunner.run(new Configuration(), new MapReduceDriver(), args);
    System.exit(res);

}



@Override
public int run(String[] args) throws Exception {

    Configuration conf =  this.getConf();
    ....

When using the -libjars I once put the required jar files on the local driver and once on the hdfs but neither worked. How can I make sure that the -libjars works?

p.s. I'm using 2-node HDInsight cluster (running in Microsoft Azure).

Here is the error message I get

 Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.microsoft.azure.documentdb.hadoop.DocumentDBInputFormat not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:174)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.ClassNotFoundException: Class com.microsoft.azure.documentdb.hadoop.DocumentDBInputFormat not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959)
    ... 8 more

Copyright License:
Author:「HHH」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/30447465/class-not-found-in-hadoop-job

About “Class not found in Hadoop job” questions

I have a map reduce job which gets its input from DocumentDB. I've added to jar files under the lib directory in my source code and also user the -libjars when running the job. but I still get the ...
I am trying to submit a mapreduce job from eclipse to a jobtracker (in this case its running on local machine) Here is the code package org.myorg; import org.apache.hadoop.conf.Configuration; im...
I have developed a map-reduce program using Apache Hadoop 1.2.1. I did the initial development using the Eclipse IDE to simulate the hadoop distributed computing environment with all the input and
I'm following example of the book "Hadoop: The Definitive Guide 2/e". I encounter a problem..:-(. I used ubuntu 12.04, hadoop 2.2.0. I made the job.jar using eclipse. The class map_reduce.program...
Hi I'm having trouble and I haven't been able to get help yet from similar threads. I am doing an example of a hadoop job and I'm just trying to run it from the IDE right now. Here is my source c...
I am trying to run a map-reduce job using spring data appache hadoop with hadoop version 0.20.0 and spring - hadoop version 1.0. i am getting No class found exception at running map task.Please h...
I have a mapreduce job which takes an avro file as input. I export it along with all the required libraries (jar library files) into a jar file. I have 2 different clusters, one is HDInsight simula...
I'm working through the hadoop example found on this page, and am encountering an error for Class not found. Eclipse is not seeing any syntax errors, and even highlights the class TokenizerMapper w...
I am running wordcount job from eclipse on hadoop-2.6.0. While running I observed warning: No job jar file set. User classes may not be found. See Job or Job#setJar(String) and when program sta...
Exception in thread "main" java.lang.ClassNotFoundException:WordCount-> so many answers relate to this issue and it seems like I am definitely missing a small point again which took me hours to fig...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.