Running of java program on hadoop-cluster with maven

2014-07-18T00:37:06

I can execute java program by build it with maven on hadoop-cluster.

source-code of my program

package com.ait.summer.clusterermapreduce;

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.mahout.clustering.canopy.CanopyDriver;
import org.apache.mahout.clustering.kmeans.KMeansDriver;
import org.apache.mahout.common.HadoopUtil;
import org.apache.mahout.common.distance.EuclideanDistanceMeasure;

public class ClustererMapReduce {

    public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {

        Configuration conf = new Configuration();
        FileSystem hdfs = FileSystem.get(conf);

        Path ProjectFolder = new Path (args[0]);

        Path InputDir = new Path(args[0], "in");
        Path OutputDir = new Path(args[0], "out");
        Path canopyCentroids = new Path(args[0], "canopyCentroids");
        HadoopUtil.delete(conf, OutputDir);

        double clusterClassificationThreshold = 0.5;
        double t1 = 20;
        double t2 = 30;
                // how to set up correctly param?

        CanopyDriver.run(conf, InputDir, canopyCentroids, new EuclideanDistanceMeasure(),
                t1, t2, false, clusterClassificationThreshold, false);        

        KMeansDriver.run(conf, InputDir, new Path(canopyCentroids, "clusters-0"), OutputDir, new EuclideanDistanceMeasure(), 0.5 , 5, true, 0.5 , false);

    }
}

my pom.xml file

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.ait.summer</groupId>
    <artifactId>ClustererMapReduce</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <repositories>
    <repository>
      <id>apache.snapshots</id>
      <name>Apache Snapshot Repository</name>
      <url>http://repository.apache.org/snapshots</url>
      <releases>
        <enabled>false</enabled>
      </releases>
    </repository>

    <repository>
      <id>cdh.repo</id>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
      <name>Cloudera Repositories</name>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </repository>

    <repository>
      <id>cdh.snapshots.repo</id>
      <url>https://repository.cloudera.com/artifactory/libs-snapshot-local</url>
      <name>Cloudera Snapshots Repository</name>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <releases>
        <enabled>false</enabled>
      </releases>
    </repository>
  </repositories>
    <dependencies>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.0.0-cdh4.7.0</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>org.apache.mahout</groupId>
            <artifactId>mahout-core</artifactId>
            <version>0.7-cdh4.7.0</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>jfree</groupId>
            <artifactId>jfreechart</artifactId>
            <version>1.0.8a</version>
        </dependency>
    </dependencies>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>
</project>

after compiling this code from command line (i situated in project dir)

$ mvn install

I run it with

$ hadoop jar

and received this error-log

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/common/distance/DistanceMeasure
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:201)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.common.distance.DistanceMeasure
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 3 more

I use CDH-4.7.0 JDK 1.7.0.65 use NetBeans IDE 8.0

What might be cause of this problem?

Copyright License:
Author:「ivanicki.ilia」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/24809048/running-of-java-program-on-hadoop-cluster-with-maven

About “Running of java program on hadoop-cluster with maven” questions

I can execute java program by build it with maven on hadoop-cluster. source-code of my program package com.ait.summer.clusterermapreduce; import java.io.IOException; import org.apache.hadoop.conf.
Background I am a Maven newbie and I greatly like the fact that Maven knows where to pick up all the JAR files needed for executing a Java program (i.e. the fact that the required JARs need not be
I have selenium maven project created. For running the tests I have to run one program that will create testNG.xml which contains list of test cases to be executed. But for that I need to run one J...
I am running following commands C:\Program Files (x86)\Apache Maven 3.1.0\bin>set JAVA_HOME=C:\Program Files (x86)\Java\jdk1.8.0_40 C:\Program Files (x86)\Apache Maven 3.1.0\bin>set PATH=%PATH%;%
I'm attempting to compile javabot on windows using maven 2. I've compiled java programs in the past ( so I know the default JDK is working correctly), and this is my first time using maven. The err...
I feel as though I am missing a very simple step or misunderstood something. I cant seem to run (on intellij) my java program that I wrote in my maven project. I wrote some code and added some
I am trying to run spark standalone java program using Maven but facing issues while building the project as below. Spark version: 1.1.0 mvn clean package mvn exec:java -Dexec.mainClass="SimpleAp...
I am trying to run maven from my java class based on this suggestion: How to run maven from java? Runtime.getRuntime().exec("mvn"); but I get: java.io.IOException: Cannot run program "mvn":
I have a java program that I coded in intellij using maven. I exported it as a jar and tried running from the command line but it keeps on telling me Error: Could not find or load main class com.co...
I want to run Maven command from Java in Mac but I am getting the following error: Exception in thread "main" java.io.IOException: Cannot run program "mvn": error=2, No such file or directory My ...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.