Hadoop > is datanode and namenode process working when a Hadoop job is running

2015-11-14T22:52:52

After Hadoop is started, two types o daemon processes are running. One is the daemon process called namenode on the namenode, the other is he daemon process called datanode on he datanode. I am sure that they are used when a big file from local file system is loaded to HDFS by means of "hdfs dfs" command.

But is it also used when a Hadoop MapReduce job is running? My understanding is no, but maybe it is also used during the Shuffle, when the outpu of map functions might be transfered from one datanode to another datanode.

Copyright License:
Author:「Rui」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/33709655/hadoop-is-datanode-and-namenode-process-working-when-a-hadoop-job-is-running

About “Hadoop > is datanode and namenode process working when a Hadoop job is running” questions

Hi I am trying to run hadoop on a server that has hadoop installed but I have no idea the directory where hadoop resides. The server was configure by the server admin. In order to load hadoop I u...
I tried to install Hadoop on a single node cluster (my own labtop-ubuntu 12.04). I followed this tutorial and checked it line by line two times . http://www.michael-noll.com/tutorials/running-hado...
This question doesn't refer to distributing jars in the whole cluster for the workers to use them. It refers to specifying a number of additional libraries on the client machine. To be more speci...
The following is my system configuration: windows8.1, vmware v12, 2 centOS VMs. I have installed hadoop on my VMs, one as master and the other as slave1. The services are correctly up and running o...
Hi i can't resolve my problem when running hadoop with start-all.sh rochdi@127:~$ start-all.sh /usr/local/hadoop/bin/hadoop-daemon.sh: line 62: [: localhost: integer expression expected ...
I was trying to setup a multi node cluster of hadoop michael-noll's way using two computers. When I tried to format the hdfs it showed a NullPointerException. hadoop@psycho-O:~/project/hadoop-0....
I know that this is a famous question across internet and there are tons of websites showing how to solve this issue and there are hell lots of QA on SO too. BUT NONE of them helped me and I am
Why are there two separate packages map-reduce package in Apache's hadoop package tree: org.apache.hadoop.mapred http://javasourcecode.org/html/open-source/hadoop/hadoop-1.0.3/org/apache/hadoop/...
What is the difference between Hadoop on Kubernetes and the standard Hadoop ? and what is the benefit from deploying Hadoop on Kubernetes ?
I'm got an error while installing Hadoop(2.6.5). I could not able to figure out and solve this problem. Why does this error occur? How do I fix this error? [INFO] [INFO] Reactor Summary: ...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.