Hadoop map reduce whole file input format

2015-04-17T03:55:10

I am trying to use the hadoop map reduce, but instead of mapping each line at a time in my Mapper, I would like to map a whole file at once.

So I have found these two classes (https://code.google.com/p/hadoop-course/source/browse/HadoopSamples/src/main/java/mr/wholeFile/?r=3) That suppose to help me do this.

And I got a compilation error that says :

The method setInputFormat(Class) in the type JobConf is not applicable for the arguments (Class) Driver.java /ex2/src line 33 Java Problem

I changed my Driver class to be

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.InputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;

import forma.WholeFileInputFormat;

/*
 * Driver
 * The Driver class is responsible of creating the job and commiting it.
 */
public class Driver {
    public static void main(String[] args) throws Exception {
        JobConf conf = new JobConf(Driver.class);
        conf.setJobName("Get minimun for each month");

        conf.setOutputKeyClass(IntWritable.class);
        conf.setOutputValueClass(IntWritable.class);

        conf.setMapperClass(Map.class);
        conf.setCombinerClass(Reduce.class);
        conf.setReducerClass(Reduce.class);

        // previous it was 
        // conf.setInputFormat(TextInputFormat.class);
        // And it was changed it to :
        conf.setInputFormat(WholeFileInputFormat.class);

        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf,new Path("input"));
        FileOutputFormat.setOutputPath(conf,new Path("output"));

        System.out.println("Starting Job...");
        JobClient.runJob(conf);
        System.out.println("Job Done!");
    }

}

What am I doing wrong?

Copyright License:
Author:「OrWn」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/29684747/hadoop-map-reduce-whole-file-input-format

About “Hadoop map reduce whole file input format” questions

I am trying to use the hadoop map reduce, but instead of mapping each line at a time in my Mapper, I would like to map a whole file at once. So I have found these two classes (https://code.google....
When I try to run map/reduce job on Hadoop cluster without specifying any input file I get following exception: java.io.IOException: No input paths specified in job Well, I can imagine cases when
I using Hadoop Map/Reduce using Java Suppose, I have completed a whole map/reduce job. Is there any way I could repeat the whole map/reduce part only, without ending the job. I mean, I DON'T want ...
I am trying to read a Excel file containing some data for aggregation in hadoop.The map reduce program seems to be working fine but the output produce is in a non readable format.Do I need to use any
I just started looking into the hadoop and made the wordcount example work on a cluster(two datanodes) after going through some struggles. But I have a question about Map/Reduce functionality. I r...
I have to parse PDF files , that are in HDFS in a Map Reduce Program in Hadoop. So i get the PDF file from HDFS as Input splits and it has to be parsed and sent to the Mapper Class. For implementin...
I'm writing an application in Java on Hadoop 1.1.1 (Ubuntu) that compares strings in order to find the longest common substrings. I've got both the map and reduce phases running successfully for sm...
I am writing map-reduce in python, but i have sufficient knowledge for the same in java, i am trying to run map-reduce in hadoop environment which is written in python language bin/hadoop jar con...
I'm running Apache's Hadoop, and using the grep example provided by that installation. I'm wondering why map reduce percentages show up running twice? I thought they only had to run once; which mak...
I am very new to hadoop , learned about its map/reduce functionality a bit , understands it wordcount demo , but not get the actual use of hadoop map/reduce in relate to database specific computati...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.