Dataingestion with Flume & Hadoop doesn't work


I'm using Flume 1.4.0 and Hadoop 2.2.0. When I'm starting Flume and writing to HDFS I get following Exception:

(SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(] process failed
java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(
        at Method)
        at java.lang.ClassLoader.loadClass(
        at sun.misc.Launcher$AppClassLoader.loadClass(
        at java.lang.ClassLoader.loadClass(
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(
        at java.lang.Class.privateGetPublicMethods(
        at java.lang.Class.privateGetPublicMethods(
        at java.lang.Class.getMethods(
        at sun.misc.ProxyGenerator.generateClassFile(
        at sun.misc.ProxyGenerator.generateProxyClass(
        at java.lang.reflect.Proxy.getProxyClass(
        at java.lang.reflect.Proxy.newProxyInstance(
        at org.apache.hadoop.ipc.ProtobufRpcEngine.getProxy(
        at org.apache.hadoop.ipc.RPC.getProtocolProxy(
        at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(
        at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(
        at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(
        at org.apache.hadoop.hdfs.DFSClient.<init>(
        at org.apache.hadoop.hdfs.DFSClient.<init>(
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(
        at org.apache.hadoop.fs.FileSystem.createFileSystem(
        at org.apache.hadoop.fs.FileSystem.access$200(
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(
        at org.apache.hadoop.fs.FileSystem$Cache.get(
        at org.apache.hadoop.fs.FileSystem.get(
        at org.apache.hadoop.fs.Path.getFileSystem(
        at org.apache.flume.sink.hdfs.BucketWriter.doOpen(
        at org.apache.flume.sink.hdfs.BucketWriter.access$000(
        at org.apache.flume.sink.hdfs.BucketWriter$
        at org.apache.flume.sink.hdfs.BucketWriter$
        at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(
        at org.apache.flume.sink.hdfs.BucketWriter.append(
        at org.apache.flume.sink.hdfs.HDFSEventSink$
        at org.apache.flume.sink.hdfs.HDFSEventSink$
        at java.util.concurrent.FutureTask$Sync.innerRun(
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$

The part of my hdfs-sink in the flume.conf is looking like this:

Define a sink that outputs to hdfs = memory-channel
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs.path = hdfs://localhost:8020/flume
agent.sinks.hdfs-sink.hdfs.fileType = DataStream
agent.sinks.hdfs-sink.hdfs.writeFormat = Text
agent.sinks.hdfs-sink.hdfs.rollCount = 10
agent.sinks.hdfs-sink.hdfs.batchSize = 10
agent.sinks.hdfs-sink.hdfs.rollSize = 0

I hope anyone can help me.

Copyright License:
Author:「user2991304」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:

About “Dataingestion with Flume & Hadoop doesn't work” questions

I'm using Flume 1.4.0 and Hadoop 2.2.0. When I'm starting Flume and writing to HDFS I get following Exception: (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.
From Apache Flume 1.6 Official website , I find flume is distributed. But Master-slave architecture has been deprecated after Flume 1.x. How does flume distribute the work? I have flume installed o...
I am trying to configure flume and am following this link. The following command works for me: flume-ng agent -n TwitterAgent -c conf -f /usr/lib/apache-flume-1.7.0-bin/conf/flume.conf The resul...
I am using Flume 1.6 and have a custom sink implementation. I have built a JAR file with all necessary dependencies and placed it under &lt;FLUME_DIR&gt;/plugins.d/MySink/lib/MySink.jar As far as ...
I am just starting with FLUME. Many installation guides mentioned like single node installation of FLUME… Is there a distributed flavor? How do we install that? What are collectors, i saw them in few
I've a flume memory channel and I want to know if exists a way to be sure that stopping a flume agent will not cause data loss on the channel. A possible solution could be to stop the source, attend
I have created a custom source for flume and copied the jar files in the following locations : mkdir -p /usr/lib/flume-ng/plugins.d/MyFlumeSource/lib/MyFlumeSource.jar chown -R flume:flume /var...
I am working with flume to ingest a ton of data into hdfs (about petabytes of data). I would like to know how is flume making use of its distributed architecture? I have over 200 servers and I have
I am trying to read a log file from /home/cloudera/Documents/flume/ and write it to hdfs using apache flume . I used the following command to create flumeLogTest folder in hdfs : sudo -u hdfs hado...
I am new with Apache Flume. I understand that Apache Flume can help transport data. But I still fail to see the ultimate benefit offered by Apache Flume. If I can configure a software or make a so...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.