Custom Flume Sink Deployment in Flume 1.6

2016-01-21T18:37:35

I am using Flume 1.6 and have a custom sink implementation. I have built a JAR file with all necessary dependencies and placed it under <FLUME_DIR>/plugins.d/MySink/lib/MySink.jar

As far as I can tell from reading the resources available, if I place my JAR file under the plugins.d directory the class path must be resolved automatically when I run flume-ng. However, I get the following exception stack saying that main class in my sink implementation cannot be loaded:

org.apache.flume.FlumeException: Unable to load sink type: MySink, class: MySink

QUESTION: Am I supposed to make any further specific settings in a config file in Flume? Isn't it sufficient to place the JAR file under the directory structure, that is:

<FLUME_DIR>/plugins.d/MySink/lib/MySink.jar 

Thanks....

EXCEPTION TRACE:

2016-01-21 10:52:51,500 (conf-file-poller-0) [INFO -org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:200)] Created channel kafkaChnl2
2016-01-21 10:52:51,501 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source httpSource, type http
2016-01-21 10:52:51,576 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: myownsink, type: MySink
2016-01-21 10:52:51,576 (conf-file-poller-0) [DEBUG - org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:63)] Sink type MySink is a custom type
2016-01-21 10:52:51,578 (conf-file-poller-0) [ERROR - org.apache.flume.node.
PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.
run(PollingPropertiesFileConfigurationProvider.java:142)] 

Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load sink type: 
MySink, class: MySink
at 
    org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:71)
at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:43)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:410)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(    ScheduledThreadPoolExecutor.java:180)
at  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Schedul edThreadPoolExecutor.java:294)
at  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: MySink
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at   org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)

Copyright License:
Author:「F. Aydemir」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/34921372/custom-flume-sink-deployment-in-flume-1-6

About “Custom Flume Sink Deployment in Flume 1.6” questions

I am using Flume 1.6 and have a custom sink implementation. I have built a JAR file with all necessary dependencies and placed it under &lt;FLUME_DIR&gt;/plugins.d/MySink/lib/MySink.jar As far as ...
So, I need to implement my own Flume sink. I went through this link, but my only missing part is, what do I exactly do once I am done with my Java implementation? Compile it into a .class? JAR? and...
I am generating data to a spool directory and redirecting that to a hive table using flume hive sink. Flume sink is connected with hive metastore but after that I am facing the following issue. I...
In the Flume agent I am collection the elements from Kafka topics and I need to insert them in ES. However I need to perform a previous digestion process in the sink, so I need to write a custom si...
Has anyone had success sinking data from flume to splunk? I've tried the Thrift and Avro flume sinks, but they have issues. Not really great formats for splunk, and flume keeps trying events over...
I am building a spring boot standalone application that needs to consume messages from a remote server and write them to a Syslog Server. I am using Flume embedded agent to reliably log messages. B...
I have created a custom source for flume and copied the jar files in the following locations : mkdir -p /usr/lib/flume-ng/plugins.d/MyFlumeSource/lib/MyFlumeSource.jar chown -R flume:flume /var...
We're using flume and I need to collect some log messages into rabbitmq. I found a source implementation that reads messages from rabbitmq, but I couldn't find a sink that can write messages into r...
Apologies for the newbie question, but I have a slightly quirky Flume question. Scenario: I have Sink A, which grabs data, performs filtering, then sends successful data to some HTTP service. I now
I have configured flume to read logs file and write to HDFS. When I start the flume the log files are read but it not written to HDFS. flume.log has the warning message - could not configure sink -...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.