Error while running a jar file in hadoop

2014-09-25T16:33:23

While running a jar file in hadoop, I am getting a null pointer exception. I am not able to understand what's the problem.

Following is my Driver Class:

package mapreduce;

import java.io.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;


public class StockDriver extends Configured implements Tool
{
      public int run(String[] args) throws Exception
      {
            //creating a JobConf object and assigning a job name for identification purposes
            JobConf conf = new JobConf(getConf(), StockDriver.class);
            conf.setJobName("StockDriver");

            //Setting configuration object with the Data Type of output Key and Value
            conf.setOutputKeyClass(Text.class);
            conf.setOutputValueClass(IntWritable.class);

            //Providing the mapper and reducer class names
            conf.setMapperClass(StockMapper.class);
            conf.setReducerClass(StockReducer.class);

            File in = new File(args[0]);
            int number_of_companies = in.listFiles().length;
            for(int iter=1;iter<=number_of_companies;iter++)
            {
                Path inp = new Path(args[0]+"/i"+Integer.toString(iter)+".txt");
                Path out = new Path(args[1]+Integer.toString(iter));
                //the HDFS input and output directory to be fetched from the command line
                FileInputFormat.addInputPath(conf, inp);
                FileOutputFormat.setOutputPath(conf, out);
                JobClient.runJob(conf);
            }
            return 0;
      }

      public static void main(String[] args) throws Exception
      {
            int res = ToolRunner.run(new Configuration(), new StockDriver(),args);
            System.exit(res);
      }
}

Mapper Class:

package mapreduce;

import java.io.IOException;
import gonn.ConstraintTree;

import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;

public class StockMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable>
{
      //hadoop supported data types
      private static IntWritable send;
      private Text word;

      //map method that performs the tokenizer job and framing the initial key value pairs
      public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException
      {
            //taking one line at a time and tokenizing the same
            String line = value.toString();
            String[] words = line.split(" ");
            String out = ConstraintTree.isMain(words[1]);
            word = new Text(out);

            send = new IntWritable(Integer.parseInt(words[0]));
            output.collect(word, send);
       }
}

Reducer Class:

package mapreduce;

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;

public class StockReducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable>
{
      //reduce method accepts the Key Value pairs from mappers, do the aggregation based on keys and produce the final output
      public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException
      {
            int val = 0;

            while (values.hasNext())
            {
               val += values.next().get();
            }
            output.collect(key, new IntWritable(val));
      }
}

Stack Trace:

Exception in thread "main" java.lang.NullPointerException
    at mapreduce.StockDriver.run(StockDriver.java:29)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at mapreduce.StockDriver.main(StockDriver.java:44)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

When I tried running the jar file using java -jar myfile.jar args... it is working fine. But When I tried running it on hadoop cluster using hadoop jar myfile.jar [MainClass] args... is is giving error.

Just to clarify, Line 29 is int number_of_companies = in.listFiles().length;

Copyright License:
Author:「Darshil Babel」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/26033999/error-while-running-a-jar-file-in-hadoop

About “Error while running a jar file in hadoop” questions

While running a jar file in hadoop, I am getting a null pointer exception. I am not able to understand what's the problem. Following is my Driver Class: package mapreduce; import java.io.*; imp...
Please go easy on me. I just started linux and hadoop at the same time. I have almost zero experience with linux and a complete beginner with hadoop. I downloaded the file hadoop-1.1.1-bin.tar.gz...
I am getting following error while trying to run jar through hadoop command prompt Exception in thread "main" java.io.IOException: Error opening job jar: /tmp/NewJar.jar at org.apache.hadoop.u...
I am getting the following error running a hadoop jar file Exception in thread "main" java.lang.ClassNotFoundException: hdfs: at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ...
I created maven jar file from eclipse. Getting following error while running jar. Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.
Getting jSonOject error when running hadoop jar command ERROR="Error: java\.lang\.ClassNotFoundException: org\.json\.JSONObject 15/04/01 18:44:17 INFO mapred.JobClient: Task Id :
I'm attempting to run a MapReduce job from a jar file and keep getting a ClassNotFoundException error. I'm running Hadoop 1.2.1 on a Centos 6 virtual machine. First I compiled the file exercise.j...
I am new to hadoop and have below question(s) on running hadoop jar command from edgeNode(http://www.dummies.com/programming/big-data/hadoop/edge-nodes-in-hadoop-clusters/).hadoop jar ${JAR_FILE} {
I'm trying to run the map reduce implementation of quadratic sieve algorithm on Hadoop. For this purpose I'm using karmasphere Hadoop community plugin with Netbeans. The program works fine using the
I just installed Hadoop from the yahoo developers network running on a vm. I ran the following code after start-all.sh after cd-ing to the bin folder hadoop jar hadoop-0.19.0.-examples.jar pi 10 1...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.