Spring-yarn cross-platform FsShell problems

2015-05-20T06:25:29

I'm working from the Spring YARN example hosted at GitHub. I have the basic project shell working; that is, the submission from a Windows client to a remote Hadoop Linux cluster works and the YARN container code is executed. In my container, I'm doing a simple test of FsShell as suggested in the example.

@Autowired
private Configuration configuration;

@OnContainerStart
public void containerStart() throws IOException {
  try (FsShell shell = new FsShell(configuration)) {
    for (FileStatus s : shell.ls(false, "/")) {
      log.info(s);
    }
  }
}

When the container code executes, I am getting the following error:

Application application_1427147939282_0014 failed 2 times due to AM Container for appattempt_1427147939282_0014_000002 exited with exitCode: 1 due to: Exception from container-launch.
Container id: container_1427147939282_0014_02_000001
Exit code: 1
Exception message: /bin/bash: line 0: fg: no job control
Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)

I understand that this is a well-known failure mode when submitting jobs from Windows clients to Linux and that it is related to passing the environment from Windows. This is hinted at in the YARN container environment documentation:

Be carefull if passing environment variables between different systems. For example if running a client on Windows and passing variables to Application Master running on Linux, execution wrapper in Yarn may silently fail.

In the Spring reference for configuring YARN with annotation-based configuration, it indicates how the configuration for YARN may be set using annotations. I have added the classes as indicated (see below) but as far as I can tell, the configuration is not being used. I have tried a few experiments that would throw an exception and I never see any change. I've included this code:

@Configuration
@EnableYarn(enable = Enable.CONTAINER)
public class ContainerConfiguration extends SpringYarnConfigurerAdapter {

  /** {@inheritDoc} */
  @Override
  public void configure(YarnEnvironmentConfigurer environment) throws Exception {
    super.configure(environment);
    environment.includeLocalSystemEnv(false);
  }
}

I'm new to Spring-Boot and the annotation-based configuration style. Any guidance would be appreciated. Thanks!

Copyright License:
Author:「Martin Serrano」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/30337196/spring-yarn-cross-platform-fsshell-problems

About “Spring-yarn cross-platform FsShell problems” questions

I'm working from the Spring YARN example hosted at GitHub. I have the basic project shell working; that is, the submission from a Windows client to a remote Hadoop Linux cluster works and the YARN
I am using spring for hadoop on spring boot based The only document setting FsShell only did the need But as this article An error occurred Error is: error is Could not autowire. No beans of 'Fs...
I need to implement a quick and simple hdfs get functionality in java code that works much like the command line "hadoop fs get", in that it supports wildcards. Using the Filesystem.copyToLocalFil...
I am trying to run hadoop dfs command on cygwin with Hadoop - 2.6.3 I am running following command /cygdrive/c/hadoop-2.6.4/bin/hadoop dfs -put word1 words/ which eventually throws an error java...
I'm building a wordcounter program and I want to create a working directory in the HDFS, but when I execute hdfs dfs -mkdir wordcount or other commands from hdfs dfs command list it returns me Error:
I am writing a service (daemon) that suppose to run on both *nix and Windows. The service part will have a xxxctl companion program that will need to trigger actions in service and receive a feedback
I am trying to execute below command on Hadoop hadoop fs -ls / but it is returing with the error Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/hadoop/fs/FsShell :
For my next project, the two leading frontrunners are .NET and Java. The project is to take a directory (something like My Music in Windows) that contains directories and music files (MP3s initiall...
For cross-platform development, I am not sure what is required to make Delphi code cross-platform which uses PostMessage and SendMessage calls for intra- and interprocess commuication, as there is ...
In my cross-platform Perl application, I use File::HomeDir and File::Spec for constructing platform-independent paths. How to do cross-platform globbing?

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.