Flume Distribution

2016-07-27T11:36:35

From Apache Flume 1.6 Official website , I find flume is distributed. But Master-slave architecture has been deprecated after Flume 1.x. How does flume distribute the work? I have flume installed on two hosts via Hortonworks . I feel like running 2 seperate flume on two different machines. Can someone help me understand its distribution. What if I have a HttpAgent on Host A and Host B with same source and sink configured? Will there be replication of the input data or flume co-ordinates? Kindly correct my understanding.

Copyright License:
Author:「Learner」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/38603290/flume-distribution

About “Flume Distribution” questions

From Apache Flume 1.6 Official website , I find flume is distributed. But Master-slave architecture has been deprecated after Flume 1.x. How does flume distribute the work? I have flume installed o...
I would like to ask- how can I run Apache Flume as a Linux daemon service? I know that Cloudera in CDH and Hortonworks in HDH are doing it with their third party distribution Hadoops. But how can ...
I know the Enterprise (Cloudera for example) way, by using a CM (via browser) or by Cloudera REST API one can access monitoring and configuring facilities. But how to schedule (run and rerun) flume
i have downloaded flume from "https://github.com/apache/flume/downloads"..but i am unable to build it..do i need to install thrift first in order to build flume??if so, what is the reson..i am gett...
I am creating a Flume agent that ingests data from a Kafka topic, the channel is a Kafka topic, and the sink is HDFS. The agent is supposed to take the take data from the kafka topic and land it in a
I am trying to configure flume and am following this link. The following command works for me: flume-ng agent -n TwitterAgent -c conf -f /usr/lib/apache-flume-1.7.0-bin/conf/flume.conf The resul...
I am using Flume 1.6 and have a custom sink implementation. I have built a JAR file with all necessary dependencies and placed it under <FLUME_DIR>/plugins.d/MySink/lib/MySink.jar As far as ...
I am just starting with FLUME. Many installation guides mentioned like single node installation of FLUME… Is there a distributed flavor? How do we install that? What are collectors, i saw them in few
I've a flume memory channel and I want to know if exists a way to be sure that stopping a flume agent will not cause data loss on the channel. A possible solution could be to stop the source, attend
I have created a custom source for flume and copied the jar files in the following locations : mkdir -p /usr/lib/flume-ng/plugins.d/MyFlumeSource/lib/MyFlumeSource.jar chown -R flume:flume /var...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.