Copying file from HDFS to Local Machine

2012-07-13T21:09:43

I'm having a problem with trying to "download" file from HDFS file system to my local system. (even though opposite operation works without a problem). *Note: File exists on the HDFS file system on specified path

Here is a code snippet:

    Configuration conf = new Configuration();
    conf.set("fs.defaultFS", "${NAMENODE_URI}");
    FileSystem hdfsFileSystem = FileSystem.get(conf);

    String result = "";

    Path local = new Path("${SOME_LOCAL_PATH}");
    Path hdfs = new Path("${SOME_HDFS_PATH}");

    String fileName = hdfs.getName();

    if (hdfsFileSystem.exists(hdfs))
    {
        hdfsFileSystem.copyToLocalFile(hdfs, local);
        result = "File " + fileName + " copied to local machine on location: " + localPath;
    }
    else
    {
        result = "File " + fileName + " does not exist on HDFS on location: " + localPath;
    }

    return result;

Exception that I get is following:

12/07/13 14:57:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.io.IOException: Cannot run program "cygpath": CreateProcess error=2, The system cannot find the file specified
    at java.lang.ProcessBuilder.start(Unknown Source)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
    at org.apache.hadoop.util.Shell.run(Shell.java:188)
    at org.apache.hadoop.fs.FileUtil$CygPathCommand.<init>(FileUtil.java:412)
    at org.apache.hadoop.fs.FileUtil.makeShellPath(FileUtil.java:438)
    at org.apache.hadoop.fs.FileUtil.makeShellPath(FileUtil.java:465)
    at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:573)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:565)
    at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:403)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:452)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:774)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:755)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:654)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:259)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:232)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:183)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1837)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1806)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1782)
    at com.hmeter.hadoop.hdfs.hdfsoperations.HdfsOperations.fileCopyFromHdfsToLocal(HdfsOperations.java:75)
    at com.hmeter.hadoop.hdfs.hdfsoperations.HdfsOperations.main(HdfsOperations.java:148)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    ... 22 more

Any idea what could be an issue? Why it is requiring the cyqpath for Cygwin? I'm running this code on Windows 7.

Thanks

Copyright License:
Author:「Bakir Jusufbegovic」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/11471176/copying-file-from-hdfs-to-local-machine

About “Copying file from HDFS to Local Machine” questions

I want to copy a file directly from my local Linux machine to a remote HDFS cluster i.e., I don't want to copy the file to the remote machine and then move it to HDFS using copyfromlocal command. ...
I'm having a problem with trying to "download" file from HDFS file system to my local system. (even though opposite operation works without a problem). *Note: File exists on the HDFS file system on
Hi I am running this command on one of the end node of the hadoop cluster. I want to copy the data from hdfs to some remote machine. I can't store it locally because of limited disc quota in local
There is a Linux VM with Hadoop installed and running. And there is Java app running in Eclipse that retrieve data from HDFS. If I am copying file(s) to or from HDFS inside the VM everything works ...
I am trying to find some tips to solve my problem. But I really have no idea how to do it. I have a HADOOP HDFS and I need to write data from my local machine to HDFS using Java. Currently, I've ...
I am trying to find out a solution for copying a file from my local windows machine or local linux machine to hdfs on cloud. I know one solution, remote -&gt; cluster node (using scp) node -&gt; h...
I'm dealing with kind of a bizarre use case where I need to make sure that File A is local to Machine A, File B is local to Machine B, etc. When copying a file to HDFS, is there a way to control w...
i'm trying to write python script for copying file to hdfs. i'm working on ubuntu and installed hadoop and pydoop. The following code is my script: import pydoop.hdfs as hdfs class COPYTOHDFS(): ...
Trying to copy files from hdfs to local machine using copyToLocal with the following command: Hadoop fs -copyToLocal remote path(hdfs file) destinationPath (my local path) But I am getting the
The problem I'm encountering is this: Having already put my input.txt (50MBytes) file into HDFS, I'm running python ./test.py hdfs:///user/myself/input.txt -r hadoop --hadoop-bin /usr/bin/hadoop ...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.