Hadoop Nodemanager not working in windows

2021-06-18T14:18:35

I have installed single node cluster of Hadoop on my windows PC. All the services are working fine except for nodemanager. It is throwing error and shutting down. Here is the error:

2021-06-18 11:00:33,907 ERROR nodemanager.NodeManager: Error starting NodeManager java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;

Full error:

2021-06-21 11:53:09,859 ERROR nodemanager.NodeManager: Error starting NodeManager java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat; at org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.getStat(NativeIO.java:460) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNativeIO(RawLocalFileSystem.java:821) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:735) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:703) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1432) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDirs(ResourceLocalizationService.java:1402) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceInit(ResourceLocalizationService.java:260) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:323) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:516) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:974) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054) 2021-06-21 11:53:09,863 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService failed in state STOPPED java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106) at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097) at java.util.Collections$SetFromMap.remove(Collections.java:5514) at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.deregisterDirsChangeListener(DirectoryCollection.java:221) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.deregisterLocalDirsChangeListener(LocalDirsHandlerService.java:271) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStop(ResourceLocalizationService.java:426) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStop(ContainerManagerImpl.java:720) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:526) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-06-21 11:53:09,864 WARN service.CompositeService: When stopping the service org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106) at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097) at java.util.Collections$SetFromMap.remove(Collections.java:5514) at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.deregisterDirsChangeListener(DirectoryCollection.java:221) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.deregisterLocalDirsChangeListener(LocalDirsHandlerService.java:271) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStop(ResourceLocalizationService.java:426) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStop(ContainerManagerImpl.java:720) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:526) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-06-21 11:53:09,865 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state STOPPED java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106) at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097) at java.util.Collections$SetFromMap.remove(Collections.java:5514) at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.deregisterDirsChangeListener(DirectoryCollection.java:221) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.deregisterLocalDirsChangeListener(LocalDirsHandlerService.java:271) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStop(ResourceLocalizationService.java:426) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStop(ContainerManagerImpl.java:720) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:526) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-06-21 11:53:09,865 WARN service.CompositeService: When stopping the service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106) at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097) at java.util.Collections$SetFromMap.remove(Collections.java:5514) at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.deregisterDirsChangeListener(DirectoryCollection.java:221) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.deregisterLocalDirsChangeListener(LocalDirsHandlerService.java:271) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStop(ResourceLocalizationService.java:426) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStop(ContainerManagerImpl.java:720) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:526) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-06-21 11:53:09,866 INFO service.AbstractService: Service NodeManager failed in state STOPPED java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106) at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097) at java.util.Collections$SetFromMap.remove(Collections.java:5514) at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.deregisterDirsChangeListener(DirectoryCollection.java:221) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.deregisterLocalDirsChangeListener(LocalDirsHandlerService.java:271) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStop(ResourceLocalizationService.java:426) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStop(ContainerManagerImpl.java:720) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:526) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-06-21 11:53:09,867 WARN service.AbstractService: When stopping the service NodeManager java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106) at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097) at java.util.Collections$SetFromMap.remove(Collections.java:5514) at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.deregisterDirsChangeListener(DirectoryCollection.java:221) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.deregisterLocalDirsChangeListener(LocalDirsHandlerService.java:271) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStop(ResourceLocalizationService.java:426) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStop(ContainerManagerImpl.java:720) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:526) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67) at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-06-21 11:53:09,867 INFO nodemanager.NodeManager: SHUTDOWN_MSG:

This error goes on and on but that's the main part of it causing the error. What can be a way around?

Copyright License:
Author:「sheharbano」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/68030237/hadoop-nodemanager-not-working-in-windows

About “Hadoop Nodemanager not working in windows” questions

I have installed single node cluster of Hadoop on my windows PC. All the services are working fine except for nodemanager. It is throwing error and shutting down. Here is the error: 2021-06-18 11:...
I have followed all the steps from this website. On my PC all are running perfectly but on my laptop nodeManager is shutting down. https://muhammadbilalyar.github.io/blogs/How-to-install-Hadoop-on-
I am setting up a single node cluster. I am following the https://dev.to/awwsmm/installing-and-running-hadoop-and-spark-on-ubuntu-18-393h tutorial. However, My nodemanager is not starting. 14468
In my Windows 10 environment, I installed Hadoop 2.10.1 While starting the nodemanager using .\start-yarn.cmd command, I am getting following error message. The DataNode, ResourceManager and Nam...
I am getting below error while starting yarn node-manager in windows10 Yarn resource manager is running. Could anyone suggest how this issue can be fixed! Hadoop version is 2.8.0 19/10/30 ...
I have compiled the hadoop binaries on windows x64 machine.. I also built the native source in the path hadoop-2.6.0-src.tar\hadoop-2.6.0-src\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\sr...
I am working on the following versions: OS- Windows 8 Hadoop- 2.3 (Single node cluster), Pig- 0.12.1, Hive- 0.13 I could start Namenode, Datanode, ResourceManager and NodeManager using windows c...
I am trying to install hadoop 2.3.0 on windows7 using windows sdk 7.1 maven command: mvn package -Pdist,native-win -DskipTests -Dtar I receive below error: I searched for this error but I coul...
I have installed the Hadoop Cluster which is the hadoop 0.23.9 version. I install the HDFS-1943.patch and now I can start all the namenode and datanode. (start-dfs.sh is working for me) However, w...
I have setup hadoop and yarn in standalone mode for now. I am trying to start all process in yarn. All process are started except nodemanager. It is throwing jvm error everytime. [root@ip-10-100-...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.