Hadoop accepting custom arguments for Driver

2015-03-27T07:15:33

How do I pass custom arguments to my hadoop mapreduce job?

For instance, if in my driver I have:

  public static void main(String[] args) throws Exception {
    try {
        String one = args[0];
        String two = args[1];
        System.out.println(two);
        System.out.println(one);
    }
    catch (ArrayIndexOutOfBoundsException e){
        System.out.println("ArrayIndexOutOfBoundsException caught");
    }
    finally {

    }
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[2]));
    FileOutputFormat.setOutputPath(job, new Path(args[3]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
 }

After I jar the file, when I run the command:

hadoop jar str1 str2 /home/bli1/wordcount/wc.jar /user/bli1/wordcount/input /user/bli1/wordcount/testout

I get:

Not a valid JAR: /nfsdata/DSCluster/home/bli1/wordcount/str1

Copyright License:
Author:「Liondancer」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/29290580/hadoop-accepting-custom-arguments-for-driver

About “Hadoop accepting custom arguments for Driver” questions

How do I pass custom arguments to my hadoop mapreduce job? For instance, if in my driver I have: public static void main(String[] args) throws Exception { try { String one = args[0]...
I am trying to connect from a Hive Database to a collection in MongoDB using a driver (jars) provided on the wiki site. Here are the steps I did: - I created a collection in MongoDB called "Diamond"
I want to achieve the hive cli driver。。but error .. help me.. error code content: 2013-12-01 12:03:09,827 WARN util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop li...
I have setup hadoop on top of mongodb using hadoop-mongodb driver. Currently I can successfully output results from a M/R job to a mongo colection. I would like to use mahout to take advantage of s...
I know that hadoop pipes can be run using the command line interface like this. But can we call the pipes program just like we call the java based mapreduce job using driver like this example ? ED...
I'm using new Hadoop API and looking for a way to pass some parameters (few strings) to mappers. How can I do that? This solutions works for old API: JobConf job = (JobConf)getConf(); job.set("
Is it possible to read a file in distributed cache inside a Hadoop driver? I have implemented the Tool Inteface and I am using the -files to add the file to the distributed cache. Or this file wil...
Is it possible to configure hadoop streaming to read two or more input arguments at runtime for a job? For example, let's say I have a script which is executed as: my_script file1 file2 How can I
I'm running Hadoop on windows and I'm trying to submit an MRJob but it comes back with the error Found 2 unexpected arguments on the command line. (cmtle) d:\>python norad_counts.py -r hadoop --...
I want to use the custom merge driver for Git. To do this I wrote the program, which takes 5 arguments %O %A %B %L %Pas described here in the block "Defining a custom merge driver" and makes 3-way ...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.