Sqoop Installation Steps


6 Answer(s)


Hi Venugopal,

The installation of Sqoop (Sqoop 1) is simple and can be done in these very simple step

  1. Ensure that the environmental variables for your hadoop installation $HADOOP_HOME. The same value should be set to a new variable $HADOOP_COMMON_HOME and $HADOOP_MAPRED_HOME.
  2. Download the zipped archive from the sqoop apache download page.
  3. Extract the archive to a location you want
  4. Create an enviromental variable $SQOOP_HOME to be the location you extracted the archive in step 2.
  5. Then add to your environmental variable PATH the values of $SQOOP_HOME/bin

Having done the above, you can be able to call the sqoop tool from comand.

An example will be checking the version of sqoop installation by running the command sqoop version.

To sqoop to import/export data from/to any relational database to/from HDFS, follow the following simple steps

  1. Search for and download the jdbc driver for the relational database you want to walk with.
  2. copy the jar to $SQOOP_HOME/lib directory. 

You can begin to use sqoop by using the appropriate sqoop cli.


Hi Venugopal,

1. Download the sqoop

2) Untar the download file
    tar -xvzf sqoop-1.4.4.bin_hadoop-1.0.0.tar.gz

3) Copy the extracted folder into sqoop location
    sudo cp -r sqoop-1.4.4.bin_hadoop-1.0.0 /usr/local/sqoop

4) Set the paths in the bash environment
    sudo vi .bashrc
    
    export PATH=$PATH:/usr/local/sqoop/bin
    export HADOOP_HOME=/usr/local/hadoop/
    export HADOOP_COMMON_HOME=/usr/local/hadoop/
    export HADOOP_MAPRED_HOME=/usr/local/hadoop/
    export HIVE_HOME=/usr/local/hive
    export HBASE_HOME=/usr/local/hbase
    export PATH=$HIVE_HOME/bin:$PATH
    
5) Update bash
    exec bash

          
6) Add permissions
        sudo chown hadoop /usr/local/sqoop
        sudo chown hadoop /usr/local/sqoop/bin
        sudo chown hadoop /usr/local/sqoop/lib
        sudo chown hadoop /usr/local/sqoop/conf
        
7) Download mysql jdbc connector(mysql webiste) and copy into /usr/local/sqoop/lib.

 

Hope this helps.

Thanks

 

 

 


Team

thankyou so much, i successfully installed Sqoop, but when i am trying to run sqoop import command, i am getting below error

My Setup is : i have installed the Centos 6 VM on my laptop and Hadoop/Sqoop is installed on the VM, and sql server is installed on my windows

 

./sqoop import --connect jdbc:mysql://<ip address>/mydb --table user_profile --username root -P  --target-dir /dezyre/sqoop/inputs/user_profile -m 2


Warning: /usr/lib/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/lib/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/lib/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
16/09/15 08:31:09 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Enter password:


16/09/15 08:31:23 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
16/09/15 08:31:23 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
16/09/15 08:31:23 ERROR manager.SqlManager: Error executing statement: java.sql.SQLNonTransientConnectionException: Could not create connection to database server.
java.sql.SQLNonTransientConnectionException: Could not create connection to database server.
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:550)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:537)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:527)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:512)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:480)
    at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:1807)
    at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:1624)
    at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:637)
    at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:351)
    at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:224)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
    at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:744)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
    at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
    at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
    at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
    at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 69
    at com.mysql.cj.mysqla.io.Buffer.readInteger(Buffer.java:271)
    at com.mysql.cj.mysqla.io.MysqlaCapabilities.setInitialHandshakePacket(MysqlaCapabilities.java:62)
    at com.mysql.cj.mysqla.io.MysqlaProtocol.readServerCapabilities(MysqlaProtocol.java:434)
    at com.mysql.cj.mysqla.io.MysqlaProtocol.beforeHandshake(MysqlaProtocol.java:359)
    at com.mysql.cj.mysqla.io.MysqlaProtocol.connect(MysqlaProtocol.java:1359)
    at com.mysql.cj.mysqla.MysqlaSession.connect(MysqlaSession.java:132)
    at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:1754)
    ... 25 more
16/09/15 08:31:23 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

 


I think you got the jdbc url wrong 

change it from

jdbc:mysql://<ip address>/mydb

to 

jdbc:mysql://<ip address>:<port>/mydb

Regards.


Hi Michael,

 

still no luck!!! :(

any other clue?


thankyou Michael and all, it got resolved

i had to put the port 3306 and had to grant *.* permission to the user in mySQL

thanks for your support