Apache Flume /var/log/flume-ng/flume.log (Permission denied)

2014-06-12T17:34:32

I am trying to read a log file from /home/cloudera/Documents/flume/ and write it to hdfs using apache flume . I used the following command to create flumeLogTest folder in hdfs :

sudo -u hdfs hadoop fs -mkdir flumeLogTest
sudo -u hdfs hadoop fs -chown flume:flume flumeLogTest

My config file is :

agent.sources = src
agent.channels = ch
agent.sinks = snk

agent.sources.src.type = exec
agent.sources.src.command = tail -f Documents/flume/vmware-0.log
agent.sources.src.channels = ch

agent.sinks.snk.channel = ch
agent.sinks.snk.type = hdfs
agent.sinks.snk.hdfs.path = hdfs://localhost.localdomain:8020/user/hdfs/flumeLogTest
agent.sinks.snk.hdfs.fileType = DataStream
agent.sinks.snk.hdfs.writeFormat = Text

agent.channels.ch.type = memory
agent.channels.ch.capacity = 100

And I started the flume-agent by the following command :

/usr/bin/flume-ng agent --conf Documents/flume --conf-file Documents/flume/Example1.conf --name agent

where Documents/flume is a folder in /home/cloudera/ this folder contains the following file

  1. flume-conf.properties.template
  2. log4j.properties
  3. flume-env.sh.template
  4. flume-env.sh
  5. Example1.conf

and I'm getting the following error:

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/flume-ng/flume.log (Permission denied)
    at java.io.FileOutputStream.openAppend(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:192)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:116)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
    at org.apache.flume.node.Application.<clinit>(Application.java:58)

I hope I am clear with the question.Please help me to sort it out. Thanks in advance.

Copyright License:
Author:「SparkOn」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/24181033/apache-flume-var-log-flume-ng-flume-log-permission-denied

About “Apache Flume /var/log/flume-ng/flume.log (Permission denied)” questions

I am trying to read a log file from /home/cloudera/Documents/flume/ and write it to hdfs using apache flume . I used the following command to create flumeLogTest folder in hdfs : sudo -u hdfs hado...
I'm pretty new using Flume, just started testing it using a CDH 4.7.0 distributuion. I'm configuring Flume through Cloudera Manager. I've set up an agent using a sequence generator as source and
My flume.log file was getting enormous so I foolishly deleted it. It's absolute path was /etc/apache-flume-1.8.0-bin/logs/flume.log I then recreated it (ie: touch flume.log) in the same directory.
I have been working in Flume for a while and faced this problem recently. (BTW, I am currently working on flume-ng 1.3.1) When we used the flume agent start command as "flume-ng agent --conf con...
I have installed Flume for the first time. I am using hadoop-1.2.1 and flume 1.6.0 I tried setting up a flume agent by following this guide. I executed this command : $ bin/flume-ng agent -n $
I have a flume with a httpsource. When I start my agent I get permission denied error shown below. ERROR org.apache.flume.source.http.HTTPSource: Error while starting HTTPSo$ java.net.SocketExcep...
I am running a flume agent with a netcat source, file channel and mongodb sink, I want to configure log4j to work with the server as flume is running as a background process. My configuration file ...
I need help. I've downloaded Apache Flume and installed outside Hadoop, just wanna try netcat logging through console. I used 1.6.0 version. Here's my conf https://gist.github.com/ans-4175/
Just begin to learn Apache Flume. I follow the instructions on Flume official getting started website: https://cwiki.apache.org/confluence/display/FLUME/Getting+Started Almost everything is fine...
I have configured flume to read logs file and write to HDFS. When I start the flume the log files are read but it not written to HDFS. flume.log has the warning message - could not configure sink -...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.