how is flume distributed?

2014-06-20T03:11:03

I am working with flume to ingest a ton of data into hdfs (about petabytes of data). I would like to know how is flume making use of its distributed architecture? I have over 200 servers and I have installed flume in one of them from where I would get the data from (aka data source) and the sink is the hdfs. (hadoop is running over serengeti in these servers). I am not sure whether flume distributes itself over the cluster or I have installed it incorrectly. I followed apache's user guide for flume installation and this post of SO.

How to install and configure apache flume?

http://flume.apache.org/FlumeUserGuide.html#setup

I am a newbie to flume and trying to understand more about it..Any help would be greatly appreciated. Thanks!!

Copyright License:
Author:「user3371005」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/24314530/how-is-flume-distributed

About “how is flume distributed?” questions

I am just starting with FLUME. Many installation guides mentioned like single node installation of FLUME… Is there a distributed flavor? How do we install that? What are collectors, i saw them in few
I am working with flume to ingest a ton of data into hdfs (about petabytes of data). I would like to know how is flume making use of its distributed architecture? I have over 200 servers and I have
From Apache Flume 1.6 Official website , I find flume is distributed. But Master-slave architecture has been deprecated after Flume 1.x. How does flume distribute the work? I have flume installed o...
My company is considering using flume for some fairly high volume log processing. We believe that the log processing needs to be distributed, both for volume (scalability) and failover (reliability)
Iam currently working on a big data project for sentiment analysis of twitter's trending topics. I followed the tutorial of cloudera and understood how to get tweets to Hadoop through flume. http:...
We have a scenario where there will be a continuous incoming set of binary files (ASN.1 type to be exact). We want to convert these binary files to different format, say XML or JSON, and write to a
I have downloaded Latest version of Apache Flume "apache-flume-1.8.0-bin" and changed the configuration as below for it's first start.I am setting up apache flume in Windows 8.1 # Licensed to the ...
is it possible to sink data from apache flume to syslog? I am currently checking from official documention, but seems there is no sink connector to Syslog available at presence. I am thnking using
I currently have this configuration in Flume : # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this wo...
I have set up flume agents and the process is working fine. However, I am not sure how to implement error detection for production environment. Any direction/guidelines regarding error handling/

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.