Loading Hadoop-Azure in Kubernetes

2021-04-16T21:36:03

I am trying to get Apache Spark to load Hadoop-Azure when running using the new Kubernetes feature.

No matter my efforts, Apache Spark always gives me the following error java.lang.classnotfoundexception: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found when trying to load a file using the wabs:// schema

My dockerfile right now

FROM spark:latest

COPY *.jar $SPARK_HOME/jars

ENV SPARK_EXTRA_CLASSPATH="$SPARK_HOME/jars/hadoop-azure-3.2.0.jar:$SPARK_HOME/jars/azure-keyvault-core-1.2.4.jar:$SPARK_HOME/jars/azure-storage-8.6.6.jar:$SPARK_HOME/jars/azure-storage-8.6.6.jar:$SPARK_HOME/jars/jetty-util-ajax-9.3.24.v20180605.jar:$SPARK_HOME/jars/wildfly-openssl-2.1.3.Final.jar"
ENV HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-azure-datalake"

(spark:latest is a build of kubernetes\dockerfiles\spark\Dockerfile from spark-3.1.1-bin-hadoop3.2)

The directory contains the following jars:

  • hadoop-azure-3.2.0.jar
  • azure-storage-8.6.6.jar
  • azure-keyvault-core-1.2.4.jar
  • jetty-util-ajax-9.3.24.v20180605.jar
  • wildfly-openssl-2.1.3.Final.jar

I have validated that the files are copied and stored in /opt/spark/jars

Copyright License:
Author:「Nisd」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/67126256/loading-hadoop-azure-in-kubernetes

About “Loading Hadoop-Azure in Kubernetes” questions

I am trying to get Apache Spark to load Hadoop-Azure when running using the new Kubernetes feature. No matter my efforts, Apache Spark always gives me the following error java.lang.classnotfoundexc...
Hy We're trying to get our website working on kubernetes (running in a container using nginx). We use ingress to route to the site, here is our configuration: nginx-conf: server { listen 80;
I followed this Google cloud Kubernetes tutorial for python. I basically changed what's in their hello world function to plot with matplotlib (with some other functions beforehand to get data to pl...
I am using hadoop-azure to access Azure blob file system (ABFS) instance, specifically when I try to list all files/directories in a specific path, it works normally for normal directories with a l...
I am using hadoop-azure to access Azure blob file system (ABFS) instance, specifically when I try to list all files/directories in a specific path, it works normally for normal directories with a l...
I am trying to learn about Spring Cloud Kubernetes for loading secrets and what I have observed is if a property has yml like structure, then it doesn't get loaded in app. Ex: kind: Secret metadata...
i installed spinnaker using k8s follow https://github.com/spinnaker/spinnaker/tree/master/experimental/kubernetes/simple, all pods looks good, i can login but i can't create application. it just lo...
I have a ReactJS front end, Spring boot backend app deployed on a baremetal Kubernetes cluster which is running Kubernetes Ingress and requests are proxied to it by HAProxy. When visiting the URL o...
New question: I've followed the guestbook tutorial here: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/guestbook/README.md And the output of my commands match their outputs
Is it possible to use VDO on Kubernetes(with Docker containers)? As far as I know, block devices are mountable - the problem here(I think) would be loading the VDO modules into the Docker containe...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.