Latest Update made on December 15, 2016.
Pig and Hive are the two key components of the Hadoop ecosystem. What does pig hadoop or hive hadoop solve? Pig hadoop and Hive hadoop have a similar goal- they are tools that ease the complexity of writing complex java MapReduce programs. However, when to use Pig Latin and when to use HiveQL is the question most of the have developers have. Apache HIVE and Apache PIG components of the Hadoop ecosystem are briefed. If we take a look at diagrammatic representation of the Hadoop ecosystem, HIVE and PIG components cover the same verticals and this certainly raises the question, which one is better? It’s Pig vs Hive (Yahoo vs Facebook).
There is no simple way to compare both Pig and Hive without digging deep into both in greater detail as to how they help in processing large amounts of data. This post compares some of the prominent features of Pig Hadoop and Hive Hadoop to help users understand the similarities and difference between them.
Hive vs Pig
|Procedural Data Flow Language||Declarative SQLish Language|
|For Programming||For creating reports|
|Mainly used by Researchers and Programmers||Mainly used by Data Analysts|
|Operates on the client side of a cluster.||Operates on the server side of a cluster.|
|Does not have a dedicated metadata database.||Makes use of exact variation of dedicated SQL DDL language by defining tables beforehand.|
|Pig is SQL like but varies to a great extent.||Directly leverages SQL and is easy to learn for database experts.|
|Pig supports Avro file format.||Hive does not support it.|
If you would like more information about Big Data careers, please click the orange "Request Info" button on top of this page.
Hadoop technology is the buzz word these days but most of the IT professionals still are not aware of the key components that comprise the Hadoop Ecosystem. Not only this, few of the people are as well of the thought that Big Data and Hadoop are one and the same.
Just before we jump on to a detailed discussion on the key components of the Hadoop Ecosystem and try to understand the differences between them let us have an understanding on what is Hadoop and what is Big Data.
What is Big Data and Hadoop?
Generally data to be stored in the database is categorized into 3 types namely Structured Data, Semi Structured Data and Unstructured Data.
Structured Data is nothing but data that can be stored in databases, for instance, the transaction records of any online purchase that you make can be stored in a database whereas data that can only be partially stored in the database is referred to as semi structured data, for instance, the data that is present in the XML records can be stored partially in the database.
Any other form of data that cannot be categorized as Structured or semi-structured is referred to as Unstructured Data, for instance, the data from Social Networking websites or the web logs which cannot be analyzed or stored for processing in the databases are examples of unstructured data.
We generally refer to Unstructured Data as “Big Data” and the framework that is used for processing Big Data is popularly known as Hadoop.
Hadoop Ecosystem comprises of the following key components:
1) Map Reduce Framework
2) HDFS (Hadoop Distributed File System)
9) Zoo Keeper
The Hadoop EcoSystem
Image Credit: jennyxiaozhang.com/6-things-you-need-to-know-about-hadoop/
Learn Hadoop Pig and Hive Components to become a Hadoop Developer!
In this post we will discuss about the two major key components of Hadoop i.e. Hive and Pig and have a detailed understanding of the difference between Pig and Hive.
Is the battle HIVE vs PIG real? Does the pair have the same advantages and disadvantages while processing enormous amounts of data? The answer is NO, there is no HIVE vs PIG in the real world, it’s just the initial ambiguity on deciding the tool which suits the need. HIVE Query language (HiveQL) suits the specific demands of analytics meanwhile PIG supports huge data operation. PIG was developed as an abstraction to avoid the complicated syntax of Java programming for MapReduce. On the other hand HIVE QL is based around SQL, which makes it easier to learn for those who know SQL. AVRO is supported by PIG making serialization faster. When it really boils down on taking decision between Pig and Hive, the suitability of the each component for the given business logic must be considered and then the decision must be taken.
Just as there is a HIVE vs PIG, there is continued discussion on Hbase vs HIVE. This uncertainty can easily be justified by taking the representation of Hadoop ecosystem. Hbase covers more vertical than HIVE. So there is no Hbase vs HIVE. With deeper insight, HIVE uses queries which will later be converted to ensemble MapReduce technique to do operations on the database, at the same time Hbase works on the HDFS directly, although Hbase and HIVE work on structured database.
Learn Hadoop to become a Microsoft Certified Big Data Engineer.
Hive Hadoop was founded by Jeff Hammerbacher who was working with Facebook. (Click here to Tweet) When working with Facebook he realized that they receive huge amounts of data on a daily basis and there needs to be a mechanism which can store, mine and help analysis of the data. This idea to mine and analyze huge amounts of data gave birth to Hive. It is Hive that has enabled Facebook to deal with 10’s of Terabytes of Data on a daily basis with ease. (Click here to Tweet)
Hive uses SQL, Hive select, where, group by, and order by clauses are similar to SQL for relational databases. Hive lose some ability to optimize the query, by relying on the Hive optimizer.
Hive is similar to a SQL Interface in Hadoop. The data that is stored in HBase component of the Hadoop Ecosystem can be accessed through Hive. Hive is of great use for developers who are not well-versed with the MapReduce framework for writing data queries that are transformed into Map Reduce jobs in Hadoop.
We can consider Hive as a Data Warehousing package that is constructed on top of Hadoop for analyzing huge amounts of data. Hive is mainly developed for users who are comfortable in using SQL. The best thing about Hive is that it conceptualizes the complexity of Hadoop because the users need not write MapReduce programs when using Hive so anyone who is not familiar with Java Programming and Hadoop API’s can also make the best use of Hive.
We can summarize Hive as:
a) A Data Warehouse Infrastructure
b) Definer of a Query Language popularly known as HiveQL (similar to SQL)
c) Provides us with various tools for easy extraction, transformation and loading of data.
d) Hive allows its users to embed customized mappers and reducers.
What makes Hive Hadoop popular?
- Hive Hadoop provides the users with strong and powerful statistics functions.
- Hive Hadoop is like SQL, so for any SQL developer the learning curve for Hive will almost be negligible.
- Hive Hadoop can be integrated with HBase for querying the data in HBase whereas this is not possible with Pig. In case of Pig, a function named HbaseStorage () will be used for loading the data from HBase.
- Hive Hadoop has gained popularity as it is supported by Hue.
- Hive Hadoop has various user groups such as CNET, Last.fm, Facebook, and Digg and so on.
For the complete list of big data companies and their salaries- CLICK HERE
Pig Hadoop was developed by Yahoo in the year 2006 so that they can have an ad-hoc method for creating and executing MapReduce jobs on huge data sets. The main motive behind developing Pig was to cut-down on the time required for development via its multi query approach. Pig is a high level data flow system that renders you a simple language platform popularly known as Pig Latin that can be used for manipulating data and queries.
Pig is used by Microsoft, Yahoo and Google, to collect and store large data sets in the form of web crawls, click streams and search logs. Pig at times finds its usage in ad-hoc analysis and processing of information.
What makes Pig Hadoop popular?
- Pig Hadoop follows a multi query approach thus it cuts down on the number times the data is scanned.
- Pig Hadoop is very easy to learn read and write if you are familiar with SQL.
- Pig provides the users with a wide range of nested data types such as Maps, Tuples and Bags that are not present in MapReduce along with some major data operations such as Ordering, Filters, and Joins.
- Performance of Pig is on par with the performance of raw Map Reduce.
- Pig has various user groups for instance 90% of Yahoo’s MapReduce is done by Pig, 80% of Twitter’s MapReduce is also done by Pig and various other companies such as Sales force, LinkedIn, AOL and Nokia also employ Pig.
Apache Pig Use Cases -Companies Using Apache Pig
- Dataium uses Apache Pig to sort and prepare data before it is handed over to MapReduce jobs.
- Apache Pig is an integral part of the "People You May Know" data product at LinkedIn.
- PayPal is a major contributor to the Pig -Eclipse project and uses Apache Pig to analyze transactional data and prevent fraud.
Difference between Pig and Hive
The below tabular data will give you an overview on the basic difference between Pig and Hive:
|Language Name||Pig Latin||HiveQL|
|Type of Language||Dataflow||Declarative (SQL Dialect)|
|Data Structures Supported||Nested and Complex|
|Turing Complete||YES,when you extend it with Java User Defined Functions.||YES, when you extend it with Java User Defined Functions.|
Instead of writing Java code to implement MapReduce, one can opt between Pig Latin and Hive SQL languages to construct MapReduce programs. Benefit of coding in Pig and Hive is - much fewer lines of code, which reduces the overall development and testing time.
Difference between pig and hive is Pig needs some mental adjustment for SQL users to learn. Pig Latin has many of the usual data processing concepts that SQL has, such as filtering, selecting, grouping, and ordering, but the syntax is a little different from SQL (particularly the group by and flatten statements!).
Hive is commonly used at Facebook for analytical purposes. Facebook promotes the Hive language. However, Yahoo! is a big advocate for Pig Latin. Yahoo! has one of the biggest Hadoop clusters in the world. Their data engineers use Pig for data processing on their Hadoop clusters. Alternatively, you may choose one among Pig and Hive at your organization, if no standards are set.
Data engineers have better control over the dataflow (ETL) processes using Pig Latin, especially with procedural language background. A data analyst finds that one can ramp up on Hadoop faster, by using Hive, especially with previous experience of SQL. If you really want to become a Hadoop expert, then you should learn both Pig and Hive for the ultimate flexibility.
Pig vs. Hive
Depending on your purpose and type of data you can either choose to use Hive Hadoop component or Pig Hadoop Component based on the below differences :
1) Hive Hadoop Component is used mainly by data analysts whereas Pig Hadoop Component is generally used by Researchers and Programmers.
2) Hive Hadoop Component is used for completely structured Data whereas Pig Hadoop Component is used for semi structured data.
3) Hive Hadoop Component has a declarative SQLish language (HiveQL) whereas Pig Hadoop Component has a procedural data flow language (Pig Latin)
4) Hive Hadoop Component is mainly used for creating reports whereas Pig Hadoop Component is mainly used for programming.
5) Hive Hadoop Component operates on the server side of any cluster whereas Pig Hadoop Component operates on the client side of any cluster.
6) Hive Hadoop Component is helpful for ETL whereas Pig Hadoop is a great ETL tool for big data because of its powerful transformation and processing capabilities.
7) Hive can start an optional thrift based server that can send queries from any nook and corner directly to the Hive server which will execute them whereas this feature is not available with Pig.
8) Hive directly leverages SQL expertise and thus can be learnt easily whereas Pig is also SQL-like but varies to a great extent and thus it will take some time efforts to master Pig.
9) Hive makes use of exact variation of the SQL DLL language by defining the tables beforehand and storing the schema details in any local database whereas in case of Pig there is no dedicated metadata database and the schemas or data types will be defined in the script itself.
10) The Hive Hadoop component has a provision for partitions so that you can process the subset of data by date or in an alphabetical order whereas Pig Hadoop component does not have any notion for partitions though might be one can achieve this through filters.
11) Pig supports Avro whereas Hive does not.
12) Pig can be installed easily over Hive as it is completely based on shell interaction
13) Pig Hadoop Component renders users with sample data for each scenario and each step through its “Illustrate” function whereas this feature is not incorporated with the Hive Hadoop Component.
14) Hive has smart inbuilt features on accessing raw data but in case of Pig Latin Scripts we are not pretty sure that accessing raw data is as fast as with HiveQL.
15) You can join, order and sort data dynamically in an aggregated manner with Hive and Pig however Pig also provides you an additional COGROUP feature for performing outer joins.
16) Pig and Hive QL are not turing complete unless extended with Java UDF's.
17) Apache Pig is the most concise and compact language compared to Hive.
18) Hadoop Pig and Hive Hadoop outperform hand-coded Hadoop MapReduce jobs as they are optimised for skewed key distribution.
Pig vs. Hive- Performance Benchmarking
Apache Pig is usually more efficient than Apache Hive as it has many high quality codes. When implementing joins, Hive creates so many objects making the join operation slow. Here are the results of Pig vs. Hive Performance Benchmarking Survey conducted by IBM –
- Apache Pig is 36% faster than Apache Hive for join operations on datasets.
- Apache Pig is 46% faster than Apache Hive for arithmetic operations.
- Apache Pig is 10% faster than Apache Hive for filtering 10% of the data.
- Apache Pig is 18% faster than Apache Hive for filtering 90% of the data.
The results of the Hive vs. Pig Benchmarking Survey revealed Pig consistently outperformed Hive for most of the operations except for grouping of data.
To conclude with after having understood the differences between Pig and Hive, to me both Hive Hadoop and Pig Hadoop Component will help you achieve the same goals, we can say that Pig is a script kiddy and Hive comes in, innate for all the natural database developers. When it comes to access choices, Hive is said to have more features over Pig. Both the Hive and Pig components are reportedly having near about the same number of committers in every project and likely in the near future we are going to see great advancements in both on the development front.