need help with permission
7 Answer(s)
DeZyre
Hi
Like you guessed - this is a permission issue. Let me explain.
The /user/ directory is owned by "hdfs" - hdfs is a superuser
You are logging in as user cloudera - so you will have permissions only to create or write files in the folder /user/cloudera
So a simple fix to your problem would be specify the /user/cloudera folder as shown below
hadoop jar hadoop-examples.jar teragen 1000 /user/cloudera/teragen/out2
Then you can view the output as follows
hadoop fs -cat /user/cloudera/teragen/out2/part-00001
Hope this helps
Thanks
Jul 22 2014 11:27 PM
Like you guessed - this is a permission issue. Let me explain.
The /user/ directory is owned by "hdfs" - hdfs is a superuser
You are logging in as user cloudera - so you will have permissions only to create or write files in the folder /user/cloudera
So a simple fix to your problem would be specify the /user/cloudera folder as shown below
hadoop jar hadoop-examples.jar teragen 1000 /user/cloudera/teragen/out2
Then you can view the output as follows
hadoop fs -cat /user/cloudera/teragen/out2/part-00001
Hope this helps
Thanks
Robert
I am having the same issue, please help me to resolve.
I logged in as user cloudera and ran the following command as suggested: hadoop jar hadoop-examples.jar teragen 1000 /user/cloudera/teragen/out2
I received the following error:
Exception in thread "main" java.io.IOException: Error opening job jar: hadoop-examples.jar
at org.apache.hadoop.util.RunJar.main(RunJar.java:135)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:127)
at java.util.jar.JarFile.(JarFile.java:135)
at java.util.jar.JarFile.(JarFile.java:72)
at org.apache.hadoop.util.RunJar.main(RunJar.java:133)
Jul 23 2014 10:18 PM
I logged in as user cloudera and ran the following command as suggested: hadoop jar hadoop-examples.jar teragen 1000 /user/cloudera/teragen/out2
I received the following error:
Exception in thread "main" java.io.IOException: Error opening job jar: hadoop-examples.jar
at org.apache.hadoop.util.RunJar.main(RunJar.java:135)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.
at java.util.jar.JarFile.
at java.util.jar.JarFile.
at org.apache.hadoop.util.RunJar.main(RunJar.java:133)
Robert
I found 2 solutions that work as follows but would like to know what is the best solution and why:
Solution 1 - hdfs is the user that owns the folder being accessed. We can run the command "hadoop fs -mkdir /dezyre" as the hdfs user using "sudo -u hdfs" as a prefix to "hadoop fs -mkdir /dezyre" ("sudo -u hdfs hadoop fs -mkdir /dezyre"). Now the command executes without error. If we use this method consistently then it will require a conscious modification to any commands discovered in normal documentation.
Solution 2 - supergroup is the group that owns the folder being accessed, so any users in this group will be able to access the files. Would the ideal solution be to create a user group called supergroup and add all users that should access files under that ownership in HDFS to the group? Since HDFS permissions are tied into the permissions set on the local file system for any users accessing the cluster, this could be performed using the following commands (make sure to su to root):
groupadd supergroup
usermod -a -G supergroup cloudera
Could someone please let me know what the best solution is and why and if you have any other solutions that would work.
Jul 24 2014 03:33 AM
Solution 1 - hdfs is the user that owns the folder being accessed. We can run the command "hadoop fs -mkdir /dezyre" as the hdfs user using "sudo -u hdfs" as a prefix to "hadoop fs -mkdir /dezyre" ("sudo -u hdfs hadoop fs -mkdir /dezyre"). Now the command executes without error. If we use this method consistently then it will require a conscious modification to any commands discovered in normal documentation.
Solution 2 - supergroup is the group that owns the folder being accessed, so any users in this group will be able to access the files. Would the ideal solution be to create a user group called supergroup and add all users that should access files under that ownership in HDFS to the group? Since HDFS permissions are tied into the permissions set on the local file system for any users accessing the cluster, this could be performed using the following commands (make sure to su to root):
groupadd supergroup
usermod -a -G supergroup cloudera
Could someone please let me know what the best solution is and why and if you have any other solutions that would work.
Shobhan
hi Robert,
you need to provide complete path of hadoop-examples jar file execute the following in cloudera. There is no need to make an permission changes.
hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar teragen 1000 /user/cloudera/teraoutput
check the output using
hadoop dfs -cat /user/cloudera/teraoutput
Thanks
Jul 24 2014 12:18 PM
you need to provide complete path of hadoop-examples jar file execute the following in cloudera. There is no need to make an permission changes.
hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar teragen 1000 /user/cloudera/teraoutput
check the output using
hadoop dfs -cat /user/cloudera/teraoutput
Thanks
Robert
Hi Shobhanc,
Ok, I understand what you are saying regarding the hadoop jar command for the hadoop-examples.jar file.
In regards to creating a directory; if I am within a folder where cloudera owns the folder like the /user/cloudera folder, then there is no need to make permission changes to make a directory/folder. But if we want to make the /dezyre directory like in the page 2 of the HDFS command assignment, then we must change permissions since hdfs is the user that owns that folder.
My question in regards to this situation where we want to make a directory where hdfs is the user that owns the folder; What is the best practice; my Solution 1 or my Solution 2 as stated previously or something different?
Thank you,
Rob
Jul 25 2014 09:29 AM
Ok, I understand what you are saying regarding the hadoop jar command for the hadoop-examples.jar file.
In regards to creating a directory; if I am within a folder where cloudera owns the folder like the /user/cloudera folder, then there is no need to make permission changes to make a directory/folder. But if we want to make the /dezyre directory like in the page 2 of the HDFS command assignment, then we must change permissions since hdfs is the user that owns that folder.
My question in regards to this situation where we want to make a directory where hdfs is the user that owns the folder; What is the best practice; my Solution 1 or my Solution 2 as stated previously or something different?
Thank you,
Rob
Shobhan
hi Robert,
The user "cloudera" is part of the super group and when you create a directory in hdfs, all the required permissions are set. I've never modified any permissions for any created files/dierctories in hdfs.
Lets see the issue in tomorrows session.
Thanks
Jul 25 2014 11:06 AM
The user "cloudera" is part of the super group and when you create a directory in hdfs, all the required permissions are set. I've never modified any permissions for any created files/dierctories in hdfs.
Lets see the issue in tomorrows session.
Thanks