hi Aditya,

Please remove the directory in hdfs or else give another output location, MR programs by default will not overwrite existing directories.

Thanks

Skip to content
Using Gmail with screen readers
Loading…


More

1 of 45,039

Web Clip
Fool.com: The Motley Fool - 3 Ways to Save $1,000 This Weekend - 2 hours ago
error 3
Inbox
x
aditya jain

10:13 AM (1 minute ago)

[cloudera@localhost Sample1]$ pwd
/home/cloudera/Desktop/dezyre/Sample1
[cloudera@localhost Sample1]$ cd /home/cloudera/Desktop/dezyre/
[cloudera@localhost dezyre]$ ls
output Sample1 Sample1.jar
[cloudera@localhost dezyre]$ hadoop fs -ls nasdaq/input/*
Found 1 items
-rw-r--r-- 3 cloudera cloudera 51579797 2015-02-04 18:53 nasdaq/input/NASDAQ_daily_prices_A.csv
Found 1 items
-rw-r--r-- 3 cloudera cloudera 510 2015-02-04 19:53 nasdaq/input/rain.txt
[cloudera@localhost dezyre]$ pwd
/home/cloudera/Desktop/dezyre
[cloudera@localhost dezyre]$ hadoop fs -ls nasdaq/*
Found 2 items
-rw-r--r-- 3 cloudera cloudera 51579797 2015-02-04 18:53 nasdaq/input/NASDAQ_daily_prices_A.csv
-rw-r--r-- 3 cloudera cloudera 510 2015-02-04 19:53 nasdaq/input/rain.txt
[cloudera@localhost dezyre]$ hadoop jar Sample1.jar NasdaqAssignment1 nasdaq/input/NASDAQ_daily_prices_A.csv nasdaq/out12
15/02/04 21:49:35 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/02/04 21:49:36 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201502040729_0037
15/02/04 21:49:36 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost.localdomain:8020/user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost.localdomain:8020/user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv already exists
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:117)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1017)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:950)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1412)
at com.assignments.NasdaqAssignment1.main(NasdaqAssignment1.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
[cloudera@localhost dezyre]$

hi Aditya,

Please let us know the complete path of nasdaq input file in hdfs.

Thanks

Its

nasdaq/input/NASDAQ_daily_prices_A.csv

it there in the above exception log.

Regards
Aditya

[cloudera@localhost dezyre]$ hadoop fs -ls nasdaq/input/*
Found 1 items
-rw-r--r-- 3 cloudera cloudera 51579797 2015-02-04 18:53 nasdaq/input/NASDAQ_daily_prices_A.csv
Found 1 items
-rw-r--r-- 3 cloudera cloudera 12225638 2015-02-04 22:01 nasdaq/input/Sample1.jar
Found 1 items
-rw-r--r-- 3 cloudera cloudera 510 2015-02-04 19:53 nasdaq/input/rain.txt
[cloudera@localhost dezyre]$

my contents of Jar file

[cloudera@localhost dezyre]$ jar tvf Sample1.jar
72 Wed Feb 04 13:00:42 PST 2015 META-INF/MANIFEST.MF
1717 Sat Apr 12 19:13:56 PDT 2014 pom.xml
562 Sat Apr 12 19:13:40 PDT 2014 .project
97 Sat Apr 12 13:30:50 PDT 2014 target/maven-archiver/pom.properties
54238 Sat Apr 12 13:30:50 PDT 2014 target/sample-1.0.jar
1433 Sat Apr 12 10:39:30 PDT 2014 .classpath
51579797 Fri Apr 11 18:57:38 PDT 2014 data/NASDAQ_daily_prices_A.csv
93575 Fri Apr 11 18:57:38 PDT 2014 data/NASDAQ_dividends_A.csv
674570 Fri Apr 11 19:49:40 PDT 2014 data/pg20417.txt
510 Fri Apr 11 18:57:38 PDT 2014 data/rain.txt
2438 Wed Feb 04 12:15:42 PST 2015 com/assignments/USPatentAssignment5$Map.class
2242 Wed Feb 04 12:15:42 PST 2015 com/assignments/USPatentAssignment5$Reduce.class
1778 Wed Feb 04 12:15:42 PST 2015 com/assignments/USPatentAssignment5.class
3710 Wed Feb 04 12:15:42 PST 2015 com/assignments/NasdaqAssignment4$Map.class
2410 Wed Feb 04 12:15:42 PST 2015 com/assignments/NasdaqAssignment4$Reduce.class
2036 Wed Feb 04 12:15:42 PST 2015 com/assignments/NasdaqAssignment4.class
3710 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment2$Map.class
2410 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment2$Reduce.class
1997 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment2.class
3455 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment3$Map.class
2232 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment3$Reduce.class
1995 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment3.class
2535 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment1$Map.class
2232 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment1$Reduce.class
1729 Wed Feb 04 12:15:44 PST 2015 com/assignments/NasdaqAssignment1.class
2541 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCountWithCombiner$Map.class
2226 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCountWithCombiner$Reduce.class
1781 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCountWithCombiner.class
3363 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCountCounters$Map.class
2210 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCountCounters$Reduce.class
1162 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCountCounters$Start.class
1805 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCountCounters.class
2493 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCount$Map.class
2178 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCount$Reduce.class
1815 Wed Feb 04 12:15:40 PST 2015 com/sample/WordCount.class
2553 Wed Feb 04 12:15:38 PST 2015 com/sample/WordCountWithPartitioner$Map.class
1474 Wed Feb 04 12:15:38 PST 2015 com/sample/WordCountWithPartitioner$MyPartitioner.class
2238 Wed Feb 04 12:15:38 PST 2015 com/sample/WordCountWithPartitioner$Reduce.class
1967 Wed Feb 04 12:15:38 PST 2015 com/sample/WordCountWithPartitioner.class
2565 Wed Feb 04 12:15:38 PST 2015 com/sample/WordCountWithoutPartitioner$Map.class
1486 Wed Feb 04 12:15:38 PST 2015 com/sample/WordCountWithoutPartitioner$MyPartitioner.class
2250 Wed Feb 04 12:15:38 PST 2015 com/sample/WordCountWithoutPartitioner$Reduce.class
1946 Wed Feb 04 12:15:38 PST 2015 com/sample/WordCountWithoutPartitioner.class
[cloudera@localhost dezyre]$

the Code runs fine if i dont use the data file from hdfs. So i am sure all the req classes are present
my jar file is on my local FS

cloudera@localhost dezyre]$ pwd
/home/cloudera/Desktop/dezyre

this is also my current dir when i try to run jar file using hadoop jar .....

hi Aditya,

Please let us know the complete path of nasdaq input file in hdfs.

Thanks

That's the path I mentioned.

Can you let me know the command that I should execute to fetch the path you are looking for.

Regards
Aditya

for command : hadoop fs -lsr / i get as follows.

drwxr-xr-x - cloudera cloudera 0 2015-02-04 18:46 /user/cloudera/nasdaq
drwxr-xr-x - cloudera cloudera 0 2015-02-04 22:01 /user/cloudera/nasdaq/input
-rw-r--r-- 3 cloudera cloudera 51579797 2015-02-04 18:53 /user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv

Regards
Aditya

I have in the java code as follows:

FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));

[cloudera@localhost dezyre]$ hadoop jar Sample1.jar NasdaqAssignment1 /user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv /user/cloudera/nasdaq/out123
15/02/05 02:41:28 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/02/05 02:41:30 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201502040729_0046
15/02/05 02:41:30 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost.localdomain:8020/user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost.localdomain:8020/user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv already exists
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:117)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1017)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:950)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1412)
at com.assignments.NasdaqAssignment1.main(NasdaqAssignment1.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

i am not sure why is the code saying input path exists. Input should exist and output not. I am not sure what i am doing wrong.

Regards
Aditya

hi Aditya,

can you provide execute permissions for the NASDAQ file in hdfs location. (hdfs dfs -chmod u+x
looking at the errors, one of the file is causing the Privileged exception:

15/02/05 02:41:30 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost.localdomain:8020/user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost.localdomain:8020/user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv already exists

thanks

I did that still same error:

[cloudera@localhost dezyre]$ hdfs dfs -chmod u+x nasdaq/input/
[cloudera@localhost dezyre]$ hadoop jar module3-dz.jar com.assignments.NasdaqAssignment1 nasdaq/input/NASDAQ_daily_prices_A.csv nasdaq/out12
15/02/05 03:05:56 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/02/05 03:05:57 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201502040729_0055
15/02/05 03:05:57 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost.localdomain:8020/user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost.localdomain:8020/user/cloudera/nasdaq/input/NASDAQ_daily_prices_A.csv already exists
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:117)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1017)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:950)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1412)
at com.assignments.NasdaqAssignment1.main(NasdaqAssignment1.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
[cloudera@localhost dezyre]$

The code run fine in the eclipse ide.

Regards
Aditya

i added permisson to file in before the command.