sqoop import using orcfile format



0

1) How do i import table in orc file formate into HDFS. please explain it with sqoop script .  

2) 1) How do i import table in orc file formate into direct HIVE TABLE . please explain it with sqoop script .  


3 Answer(s)


0

Hi Rakesh,

At least in Sqoop 1.4.5 there exists hcatalog integration that support orc file format (amongst others).

For example you have the option

--hcatalog-storage-stanza

which can be set to

stored as orc tblproperties ("orc.compress"="SNAPPY")

Example:

sqoop import 
 --connect jdbc:postgresql://foobar:5432/my_db 
 --driver org.postgresql.Driver 
 --connection-manager org.apache.sqoop.manager.GenericJdbcManager 
 --username foo 
 --password-file hdfs:///user/foobar/foo.txt 
 --table fact 
 --hcatalog-home /usr/hdp/current/hive-webhcat 
 --hcatalog-database my_hcat_db 
 --hcatalog-table fact 
 --create-hcatalog-table 
 --hcatalog-storage-stanza 'stored as orc tblproperties ("orc.compress"="SNAPPY")'

 

Hope this helps.


0

when i run this  scoop script 

sqoop import \

--connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" \

 --username retail_dba \

 --password cloudera \

 --table departments \

 --hcatalog-home  /user/hive/warehouse/retail_ods.db/departments \

 --hcatalog-database retail_db \

 --hcatalog-table departments \

 --create-hcatalog-table \

 --hcatalog-storage-stanza 'stored as orc tblproperties ("orc.compress"="SNAPPY")'

 

 

I am getting this error:

 

 

[cloudera@quickstart ~]$ sqoop import \

> --connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" \

>  --username retail_dba \

>  --password cloudera \

>  --table departments \

>  --hcatalog-home  /user/hive/warehouse/retail_ods.db/departments \

Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

18/04/27 00:02:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.0

18/04/27 00:02:54 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

18/04/27 00:02:54 ERROR tool.BaseSqoopTool: Error parsing arguments for import:

18/04/27 00:02:54 ERROR tool.BaseSqoopTool: Unrecognized argument: 

 

Try --help for usage instructions.

[cloudera@quickstart ~]$  --hcatalog-database retail_db \

>  --hcatalog-table departments \

>  --create-hcatalog-table \


0

Hi Rakesh,

To properly use the Hcatalog, please refer the tutorial link:

https://www.tutorialspoint.com/hcatalog/hcatalog_introduction.htm

For installation, please refer the below:

https://www.tutorialspoint.com/hcatalog/hcatalog_installation.htm

For your error, enter the following command:

export HCAT_HOME = $HiVE_HOME/HCatalog

Hope this helps.

Thanks.

Your Answer

Click on this code-snippet-icon icon to add code snippet.

Upload Files (Maximum image file size - 1.5 MB, other file size - 10 MB, total size - not more than 50 MB)

Email
Password