what is the different between configuration files under /etc/hadoop/conf and /etc/hadoop/conf.cloudera.HDFS and /etc/hadoop/conf.cloudera.YARN

2017-08-29T05:16:16

I have cloudera 5.7, I have Cloudera Manager too.

under the directory /etc/hadoop, I saw three sub-directories. /etc/hadoop/conf /etc/hadoop/conf.cloudera.HDFS/ /etc/hadoop/conf.cloudera.YARN/

the hadoop-env.sh in ../conf/ is different from ../conf.cloudera.HDFS/.. the core-site.xml in ../conf/ is different from ../conf.cloudera.HDFS/.. as well.

the hadoop-env.sh in ../conf/ has settings on YARN, while the one under../conf.cloudera.HDFS doesn't has it.

and the one in ../conf.cloudera.HDFS/.. has the settings for Namenode, datanodes, etc.

I have CM installed, I am wondering if these configuration files are really in use?

If yes, and I need to change some environment variables, should I change all of these hadoop-env.sh? and copy it to the other nodes?

Thanks.

Copyright License:
Author:「sparkmeow」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/45927493/what-is-the-different-between-configuration-files-under-etc-hadoop-conf-and-et

About “what is the different between configuration files under /etc/hadoop/conf and /etc/hadoop/conf.cloudera.HDFS and /etc/hadoop/conf.cloudera.YARN” questions

I have cloudera 5.7, I have Cloudera Manager too. under the directory /etc/hadoop, I saw three sub-directories. /etc/hadoop/conf /etc/hadoop/conf.cloudera.HDFS/ /etc/hadoop/conf.cloudera...
Any time I try to run a Spark application on a Cloudera CDH 5.4.4 cluster, Yarn client mode, I get the following exception (repeated many times in the stack trace). The process continue anyway (it ...
I'm setting up the namenode for Hadoop on a machine node1.homer. The content of /etc/hosts contains: 10.0.0.2 node1.homer node1 and /etc/resolv.conf contains: nameserver 8.8.8.8 search local...
I'm starting an application which loads creates a new YarnConfiguration() object. When I'm running it I'm setting HADOOP_CONF_DIR to /etc/hadoop/conf where the configuration files are. I'm then
While installing hadoop in pseudo distributed mode i was intially getting below error Starting namenodes on [localhost] localhost: namenode running as process 20678. Stop it first. localhost: dat...
I am new to Hadoop. I am trying to setup a single-node cluster. I have noticed that in the documentation i've read (even on Apache's config site) it always refers to the configuration files in co...
I am new to EMR and Spark. I am going through this steps mentioned here https://aws.amazon.com/premiumsupport/knowledge-center/emr-submit-spark-job-remote-cluster/ . In step#5 it says Copy all file...
I have just installed hadoop 2.8.1 on my ubuntu 14 environment. After Installation I got these dirs and files under hadoop installation folder Hadoop installation folder is /usr/local/hadoop/hadoo...
I'm trying to compile/install/run Hadoop as a single node cluster on a Mac OS X 10.7.5. I've downloaded the hadoop-2.2.0-src, and am able to compile all modules with mvn install The install is
public void readFile(String file) throws IOException { Cofiguration conf = new Configuration(); conf.addResource(new Path("/usr/local/hadoop-2.7.3/etc/hadoop/core-site.xml")) conf.addRe...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.