Cloudera Hadoop : File reading/ writing in HDFS

2017-01-09T20:34:34

I have this scala and Java code running in Spark on Cloudera platform whose simple task is to perform Word count on the files in HDFS. My question is : What's the difference in reading the file with this code snippet -

sc.textFile("hdfs://quickstart.cloudera:8020/user/spark/InputFile/inputText.txt")

as opposed to reading from local drive over cloudera platform?

sc.textFile("/home/cloudera/InputFile/inputText.txt")

Is it not that in both cases the file is saved using HDFS and wouldn't make any difference reading/ writing either ways? These both read/write to HDFS, right? I referred this thread, but no clue. Cloudera Quickstart VM illegalArguementException: Wrong FS: hdfs: expected: file:

Could you please tell me at least a single case where using hdfs:// implies something else?

Thank You!

Copyright License:
Author:「Ashwini」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/41548201/cloudera-hadoop-file-reading-writing-in-hdfs

About “Cloudera Hadoop : File reading/ writing in HDFS” questions

I have this scala and Java code running in Spark on Cloudera platform whose simple task is to perform Word count on the files in HDFS. My question is : What's the difference in reading the file wit...
I am trying to create a folder in hdfs hadoop file system but it is not allowing me to create a folder using the user cloudera nor as root. What should I configure to make it to allow me to hier wa...
So I installed Hadoop via Cloudera Manager cdh3u5 on CentOS 5. When I run cmd hadoop fs -ls / I expected to see the contents of hdfs://localhost.localdomain:8020/ However, it had returned the
I have a Hadoop Cluster in Cloudera with 4 nodes, 1 master and 3 slave, and a replication factor of 3 and within a days my cluster doesn't stop to get bigger without any reason , i don't execute an...
I have cloudera 5.7, I have Cloudera Manager too. under the directory /etc/hadoop, I saw three sub-directories. /etc/hadoop/conf /etc/hadoop/conf.cloudera.HDFS/ /etc/hadoop/conf.cloudera...
Installed Cloudera VM and loaded data file into HDFS. The command "hadoop fs -ls / " help me to verify the data in hdfs. But want to know the physical location where it is residing ? any suggestion
I use a MAC OS X Yosemite with a VM cloudera-quickstart-vm-5.4.2-0-virtualbox. When I type "hdfs dfs -put testfile.txt" to put a TEXT FILE into HDFS I get a DataStreamer Exception. I notice that th...
The scope is to read from HDFS, filter in Spark and write results to Cassandra. I am packaging and running with SBT. Here is the problem: Reading from HDFS to Spark requires the following line in ...
I have written a class(es) that writes and reads from hdfs. Given certain conditions that are occurring when these classes are instantiated they create a specific path and file, and write to it (o...
I have written my first map reduce program and when I ran it in eclipse it writes to the output file and works as expected. However when I run it from command line using hadoop jar myjar.jar the

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.