Issues Spark installation in Linux



0

I had already installed Hadoop on Ubuntu in the Hadoop course. There are 2 errors that I am getting during the installation of Spark on Linux. 

1. When I try to change the user name of the spark durectory, I am getting the following error. 

When I run this command - sudo chown hadoop /opt/spark I get the following error:

chown: invalid user: ‘hadoop’ 

2. I am setting the path for some env. variables per the instructions but the following directories are non-existent. What do you suggest?

export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop

Can you please resolve the above issues?


7 Answer(s)


0

Hi Lokesh,

1. Please check if the username is correct by using the following command:
$ whoami

It will return the current user name.


2. You can check whether the following directory available or not, use the following command:
$ ls -l /usr/local/hadoop/etc/hadoop
If you can see the contents of the hadoop file in the output, that means the directory exists.

Thanks,

Dezyre Support


0

1. Why are we changing the spark directory owner name to hadoop?

My user name is not hadoop. it is User.

$ id -un
user

2. I already mentioned that the following directory path does not exist. What do you suggest?

/usr/local/hadoop/etc/hadoop

btw, the .bashrc file is not in the directory that is the notes. It is my home directory: /home/user

 


0

These are sub-directories under /usr/local directory:

bin  etc  games  include  lib  man  sbin  share  src


0

Hi Lokesh,

The user name doesn't matter, in chown command you need to mention user instead of hadoop.

The directory should include hadoop installation directory. You need to install Hadoop first before installing Spark. If you hadoop installed in different directory. Please mention the same path for HADOOP_CONF_DIR and YARN_CONF_DIR.
 

Hope this helps.
Thanks.


0

Here's the $HADOOP_HOME:

export HADOOP_HOME=/home/user/hadoop-2.7.3

i setup the following variables to HADOOP_HOME:

export HADOOP_CONF_DIR= /home/user/hadoop-2.7.3
export YARN_CONF_DIR=/home/user/hadoop-2.7.3
 


0

Spark is installed but I am not sure if the hadoop setting are correct. When I started the spark-shell, I got some errors related to hadoop. DO yo have any suggestions?

    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
    ... 75 more
Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.


<console>:16: error: not found: value sqlContext
         import sqlContext.implicits._
                ^
<console>:16: error: not found: value sqlContext
         import sqlContext.sql


0

Hi Lokesh,

Because Hive is not setup, you are getting following error, you can ignore this:

 at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
    ... 75 more
Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.

This is sqlContext library dependecy warning, can be ignored.

<console>:16: error: not found: value sqlContext
         import sqlContext.implicits._
                ^
<console>:16: error: not found: value sqlContext
         import sqlContext.sql

Regards,

Dezyre Support