With the help of ProjectPro’s Hadoop Instructors, we have put together a detailed list of big data Hadoop interview questions based on the different components of the Hadoop Ecosystem such as MapReduce, Hive, HBase, Pig, YARN, Flume, Sqoop, HDFS, etc.
These are Hadoop Basic Interview Questions and Answers for freshers and experienced.
1. What is the difference between Hadoop and Traditional RDBMS?
|Datatypes||Processes semi-structured and unstructured data.||Processes structured data.|
|Schema||Schema on Read||Schema on Write|
|Best Fit for Applications||Data discovery and Massive Storage/Processing of Unstructured data.||Best suited for OLTP and complex ACID transactions.|
|Speed||Writes are Fast||Reads are Fast|
2. What do the four V’s of Big Data denote?
IBM has a nice, simple explanation for the four critical features of big data:
a) Volume –Scale of data
b) Velocity –Analysis of streaming data
c) Variety – Different forms of data
d) Veracity –Uncertainty of data
Here is an explanatory video on the four V’s of Big Data
3. How big data analysis helps businesses increase their revenue? Give example.
Big data analysis is helping businesses differentiate themselves – for example Walmart the world’s largest retailer in 2014 in terms of revenue - is using big data analytics to increase its sales through better predictive analytics, providing customized recommendations and launching new products based on customer preferences and needs. Walmart observed a significant 10% to 15% increase in online sales for $1 billion in incremental revenue. There are many more companies like Facebook, Twitter, LinkedIn, Pandora, JPMorgan Chase, Bank of America, etc. using big data analytics to boost their revenue.
4. Name some companies that use Hadoop.
Yahoo (One of the biggest user & more than 80% code contributor to Hadoop)
Click on this link to view a detailed list of some of the top companies using Hadoop.
5. Differentiate between Structured and Unstructured data.
Data that can be stored in traditional database systems in the form of rows and columns, for example, the online purchase transactions can be referred to as Structured Data. Data that can be stored only partially in traditional database systems, for example, data in XML records can be referred to as semi-structured data. Unorganized and raw data that cannot be categorized as semi-structured or structured data is referred to as unstructured data. Facebook updates, tweets on Twitter, Reviews, weblogs, etc. are all examples of unstructured data.
Structured data: Schema-based data, Datastore in SQL, Postgresql databases etc
Semi-structured data : Json objects , json arrays, csv , txt ,xlsx files,web logs ,tweets etc
Unstructured data : Audio, Video files, etc
6. On what concept the Hadoop framework works?
Hadoop Framework works on the following two core components-
1)HDFS – Hadoop Distributed File System is the java based file system for scalable and reliable storage of large datasets. Data in HDFS is stored in the form of blocks and it operates on the Master-Slave Architecture.
2)Hadoop MapReduce-This is a java based programming paradigm of the Hadoop framework that provides scalability across various Hadoop clusters. MapReduce distributes the workload into various tasks that can run in parallel. Hadoop jobs perform 2 separate tasks- job. The map job breaks down the data sets into key-value pairs or tuples. The reduce job then takes the output of the map job and combines the data tuples to into smaller set of tuples. The reduce job is always performed after the map job is executed.
7) What are the main components of a Hadoop Application?
Hadoop applications have wide range of technologies that provide great advantage in solving complex business problems.
Core components of a Hadoop application are-
1) Hadoop Common
3) Hadoop MapReduce
Data Access Components are - Pig and Hive
Data Storage Component is - HBase
Data Integration Components are - Apache Flume, Sqoop, Chukwa
Data Management and Monitoring Components are - Ambari, Oozie and Zookeeper.
Data Serialization Components are - Thrift and Avro
Data Intelligence Components are - Apache Mahout and Drill.
8. What is Hadoop streaming?
Hadoop distribution has a generic application programming interface for writing Map and Reduce jobs in any desired programming language like Python, Perl, Ruby, etc. This is referred to as Hadoop Streaming. Users can create and run jobs with any kind of shell scripts or executable as the Mapper or Reducers. The latest tool for Hadoop streaming is Spark.
9. What is the best hardware configuration to run Hadoop?
The best configuration for executing Hadoop jobs is dual core machines or dual processors with 4GB or 8GB RAM that use ECC memory. Hadoop highly benefits from using ECC memory though it is not low - end. ECC memory is recommended for running Hadoop because most of the Hadoop users have experienced various checksum errors by using non ECC memory. However, the hardware configuration also depends on the workflow requirements and can change accordingly.
10. What are the most commonly defined input formats in Hadoop?
The most common Input Formats defined in Hadoop are:
11. What are the steps involved in deploying a big data solution?
i) Data Ingestion – The foremost step in deploying big data solutions is to extract data from different sources which could be an Enterprise Resource Planning System like SAP, any CRM like Salesforce or Siebel , RDBMS like MySQL or Oracle, or could be the log files, flat files, documents, images, social media feeds. This data needs to be stored in HDFS. Data can either be ingested through batch jobs that run every 15 minutes, once every night and so on or through streaming in real-time from 100 ms to 120 seconds.
ii) Data Storage – The subsequent step after ingesting data is to store it either in HDFS or NoSQL database like HBase. HBase storage works well for random read/write access whereas HDFS is optimized for sequential access.
iii) Data Processing – The ultimate step is to process the data using one of the processing frameworks like mapreduce, spark, pig, hive, etc.
12. How will you choose various file formats for storing and processing data using Apache Hadoop ?
The decision to choose a particular file format is based on the following factors-
i) Schema evolution to add, alter and rename fields.
ii) Usage pattern like accessing 5 columns out of 50 columns vs accessing most of the columns.
iii)Splittability to be processed in parallel.
iv) Read/Write/Transfer performance vs block compression saving storage space
File Formats that can be used with Hadoop - CSV, JSON, Columnar, Sequence files, AVRO, and Parquet file.
CSV files are an ideal fit for exchanging data between hadoop and external systems. It is advisable not to use header and footer lines when using CSV files.
Every JSON File has its own record. JSON stores both data and schema together in a record and also enables complete schema evolution and splitability. However, JSON files do not support block level compression.
This kind of file format is best suited for long term storage with Schema. Avro files store metadata with data and also let you specify independent schema for reading the files.
A columnar file format that supports block level compression and is optimized for query performance as it allows selection of 10 or less columns from from 50+ columns records.
13. What makes Hadoop Fault tolerant?
Hadoop is said to be highly fault tolerant. Hadoop achieves this feat through the process of replication. Data is replicated across multiple nodes in a Hadoop cluster. The data is associated with a replication factor, which indicates the number of copies of the data that are present across the various nodes in a Hadoop cluster. For example, if the replication factor is 3, the data will be present in three different nodes of the Hadoop cluster, where each node will contain one copy each. In this manner, if there is a failure in any one of the nodes, the data will not be lost, but can be recovered from one of the other nodes which contains copies or replicas of the data.
Hadoop 3.0, however, makes use of the method of Erasure Coding (EC). EC is implemented by means of Redundant Array of Inexpensive Disks (RAID) by striping, where logically sequential data is divided into smaller units and these smaller units are then stored as consecutive unts on different disks. Replication results in a storage overhead of 200% in case of a replication factor of 3 (which is the default replication factor). The use of EC in Hadoop improves storage efficiency when compared to replication, but still maintains the same level of fault tolerance.
Another feature of Hadoop 3.0 which ensures more fault tolerance is that Hadoop 3.0 supports two or more Standby nodes, unlike Hadoop that supports only two NameNodes.
You have a file that contains 200 billion URLs. How will you find the first unique URL using Hadoop MapReduce?
11. What is Big Data?
Big data is defined as the voluminous amount of structured, unstructured or semi-structured data that has huge potential for mining but is so large that it cannot be processed using traditional database systems. Big data is characterized by its high velocity, volume and variety that requires cost effective and innovative methods for information processing to draw meaningful business insights. More than the volume of the data – it is the nature of the data that defines whether it is considered as Big Data or not.
12) Mention a business use case where you worked with the Hadoop Ecosystem
You can share details on how you deployed Hadoop distributions like Cloudera and Hortonworks in your organization either in a standalone environment or on the cloud. Mention how you configured the number of required nodes , tools, services, security features such as SSL, SASL, Kerberos, etc. Having set up the Hadoop cluster, talk about how you initially extracted the data from data sources like APIs, SQL based databases, etc and stored it in HDFS( storage layer) , how you performed data cleaning and validation, and the series of ETLs you performed to extract the data in the given format to extract KPIs.
Some of the ETLs tasks include :
We have further categorized Big Data Interview Questions for Freshers and Experienced-
1. What is a block and block scanner in HDFS?
Block - The minimum amount of data that can be read or written is generally referred to as a “block” in HDFS. The de
size of a block in HDFS is 64MB.
Block Scanner - Block Scanner tracks the list of blocks present on a DataNode and verifies them to find any kind of checksum errors. Block Scanners use a throttling mechanism to reserve disk bandwidth on the datanode.
2. Explain the difference between NameNode, Backup Node and Checkpoint NameNode.
NameNode: NameNode is at the heart of the HDFS file system which manages the metadata i.e. the data of the files is not stored on the NameNode but rather it has the directory tree of all the files present in the HDFS file system on a hadoop cluster. NameNode uses two files for the namespace-
fsimage file- It keeps track of the latest checkpoint of the namespace.
edits file-It is a log of changes that have been made to the namespace since checkpoint.
Checkpoint Node keeps track of the latest checkpoint in a directory that has same structure as that of NameNode’s directory. Checkpoint node creates checkpoints for the namespace at regular intervals by downloading the edits and fsimage file from the NameNode and merging it locally. The new image is then again updated back to the active NameNode.
Backup Node also provides check pointing functionality like that of the checkpoint node but it also maintains its up-to-date in-memory copy of the file system namespace that is in sync with the active NameNode.
3. What is commodity hardware?
Commodity Hardware refers to inexpensive systems that do not have high availability or high quality. Commodity Hardware consists of RAM because there are specific services that need to be executed on RAM. Hadoop can be run on any commodity hardware and does not require any super computer s or high end hardware configuration to execute jobs.
4. What is the port number for NameNode, Task Tracker and Job Tracker?
Job Tracker 50030
Task Tracker 50060
5. Explain about the process of inter cluster data copying.
HDFS provides a distributed data copying facility through the DistCP from source to destination. If this data copying is within the hadoop cluster then it is referred to as inter cluster data copying. DistCP requires both source and destination to have a compatible or same version of hadoop.
6. How can you overwrite the replication factors in HDFS?
The replication factor in HDFS can be modified or overwritten in 2 ways-
1)Using the Hadoop FS Shell, replication factor can be changed per file basis using the below command-
$hadoop fs –setrep –w 2 /my/test_file (test_file is the filename whose replication factor will be set to 2)
2)Using the Hadoop FS Shell, replication factor of all files under a given directory can be modified using the below command-
3)$hadoop fs –setrep –w 5 /my/test_dir (test_dir is the name of the directory and all the files in this directory will have a replication factor set to 5)
7. Explain the difference between NAS and HDFS.
8. Explain what happens if during the PUT operation, HDFS block is assigned a replication factor 1 instead of the default value 3.
Replication factor is a property of HDFS that can be set accordingly for the entire cluster to adjust the number of times the blocks are to be replicated to ensure high data availability. For every block that is stored in HDFS, the cluster will have n-1 duplicated blocks. So, if the replication factor during the PUT operation is set to 1 instead of the default value 3, then it will have a single copy of data. Under these circumstances when the replication factor is set to 1 ,if the DataNode crashes under any circumstances, then only single copy of the data would be lost.
9. What is the process to change the files at arbitrary locations in HDFS?
HDFS does not support modifications at arbitrary offsets in the file or multiple writers but files are written by a single writer in append only format i.e. writes to a file in HDFS are always made at the end of the file.
10. Explain about the indexing process in HDFS.
Indexing process in HDFS depends on the block size. HDFS stores the last part of the data that further points to the address where the next part of data chunk is stored.
11. What is a rack awareness and on what basis is data stored in a rack?
All the data nodes put together form a storage area i.e. the physical location of the data nodes is referred to as Rack in HDFS. The rack information i.e. the rack id of each data node is acquired by the NameNode. The process of selecting closer data nodes depending on the rack information is known as Rack Awareness.
The contents present in the file are divided into data block as soon as the client is ready to load the file into the hadoop cluster. After consulting with the NameNode, client allocates 3 data nodes for each data block. For each data block, there exists 2 copies in one rack and the third copy is present in another rack. This is generally referred to as the Replica Placement Policy.
12. What happens to a NameNode that has no data?
There does not exist any NameNode without data. If it is a NameNode then it should have some sort of data in it.
13. What happens when a user submits a Hadoop job when the NameNode is down- does the job get in to hold or does it fail.
The Hadoop job fails when the NameNode is down.
14. What happens when a user submits a Hadoop job when the Job Tracker is down- does the job get in to hold or does it fail.
The Hadoop job fails when the Job Tracker is down.
15. Whenever a client submits a hadoop job, who receives it?
NameNode receives the Hadoop job which then looks for the data requested by the client and provides the block information. JobTracker takes care of resource allocation of the hadoop job to ensure timely completion.
16. What do you understand by edge nodes in Hadoop?
Edges nodes are the interface between hadoop cluster and the external network. Edge nodes are used for running cluster adminstration tools and client applications.Edge nodes are also referred to as gateway nodes.
We have further categorized Hadoop HDFS Interview Questions for Freshers and Experienced-
Here are a few more frequently asked Hadoop HDFS Interview Questions and Answers for Freshers and Experienced
1. Explain the usage of Context Object.
Context Object is used to help the mapper interact with other Hadoop systems. Context Object can be used for updating counters, to report the progress and to provide any application level status updates. ContextObject has the configuration details for the job and also interfaces, that helps it to generating the output.
2. What are the core methods of a Reducer?
The 3 core methods of a reducer are –
1)setup () – This method of the reducer is used for configuring various parameters like the input data size, distributed cache, heap size, etc.
Function Definition- public void setup (context)
2)reduce () it is heart of the reducer which is called once per key with the associated reduce task.
Function Definition -public void reduce (Key,Value,context)
3)cleanup () - This method is called only once at the end of reduce task for clearing all the temporary files.
Function Definition -public void cleanup (context)
3. Explain about the partitioning, shuffle and sort phase
Shuffle Phase-Once the first map tasks are completed, the nodes continue to perform several other map tasks and also exchange the intermediate outputs with the reducers as required. This process of moving the intermediate outputs of map tasks to the reducer is referred to as Shuffling.
Sort Phase- Hadoop MapReduce automatically sorts the set of intermediate keys on a single node before they are given as input to the reducer.
Partitioning Phase-The process that determines which intermediate keys and value will be received by each reducer instance is referred to as partitioning. The destination partition is same for any key irrespective of the mapper instance that generated it.
4. How to write a custom partitioner for a Hadoop MapReduce job?
Steps to write a Custom Partitioner for a Hadoop MapReduce Job-
5. What are side data distribution techniques in Hadoop?
The extra read only data required by a hadoop job to process the main dataset is referred to as side data. Hadoop has two side data distribution techniques -
i) Using the job configuration - This technique should not be used for transferring more than few kilobytes of data as it can pressurize the memory usage of hadoop daemons,particularly if your system is running several hadoop jobs.
ii) Distributed Cache - Rather than serializing side data using the job configuration, it is suggested to distribute data using hadoop's distributed cache mechanism.
6. What does the jps command in hadoop do?
A. JPS is short for Java Virtual Machine Process Status Tool. it gives the status of the daemons running the Hadoop cluster. It is a tool which is used to check if Hadoop processes are running or not. JPS gives an output which mentions the status of namenode, datanode, secondary namenode, Jobtracker and Task tracker.
We have further categorized Hadoop MapReduce Interview Questions for Freshers and Experienced-
Here are a few more frequently asked Hadoop MapReduce Interview Questions and Answers
1. When should you use HBase and what are the key components of HBase?
HBase should be used when the big data application has –
1)A variable schema
2)When data is stored in the form of collections
3)If the application demands key based access to data while retrieving.
Key components of HBase are –
Region- This component contains memory data store and Hfile.
Region Server-This monitors the Region.
HBase Master-It is responsible for monitoring the region server.
Zookeeper- It takes care of the coordination between the HBase Master component and the client.
Catalog Tables-The two important catalog tables are ROOT and META.ROOT table tracks where the META table is and META table stores all the regions in the system.
2. What are the different operational commands in HBase at record level and table level?
Record Level Operational Commands in HBase are –put, get, increment, scan and delete.
Table Level Operational Commands in HBase are-describe, list, drop, disable and scan.
3. What is Row Key?
Every row in an HBase table has a unique identifier known as RowKey. It is used for grouping cells logically and it ensures that all cells that have the same RowKeys are co-located on the same server. RowKey is internally regarded as a byte array.
4. Explain the difference between RDBMS data model and HBase data model.
RDBMS is a schema based database whereas HBase is schema less data model.
RDBMS does not have support for in-built partitioning whereas in HBase there is automated partitioning.
RDBMS stores normalized data whereas HBase stores de-normalized data.
5. Explain about the different catalog tables in HBase?
The two important catalog tables in HBase, are ROOT and META. ROOT table tracks where the META table is and META table stores all the regions in the system.
6. What is column families? What happens if you alter the block size of ColumnFamily on an already populated database?
The logical deviation of data is represented through a key known as column Family. Column families consist of the basic unit of physical storage on which compression features can be applied. In an already populated database, when the block size of column family is altered, the old data will remain within the old block size whereas the new data that comes in will take the new block size. When compaction takes place, the old data will take the new block size so that the existing data is read correctly.
7. Explain the difference between HBase and Hive.
HBase and Hive both are completely different hadoop based technologies-Hive is a data warehouse infrastructure on top of Hadoop whereas HBase is a NoSQL key value store that runs on top of Hadoop. Hive helps SQL savvy people to run MapReduce jobs whereas HBase supports 4 primary operations-put, get, scan and delete. HBase is ideal for real time querying of big data where Hive is an ideal choice for analytical querying of data collected over period of time.
8. Explain the process of row deletion in HBase.
On issuing a delete command in HBase through the HBase client, data is not actually deleted from the cells but rather the cells are made invisible by setting a tombstone marker. The deleted cells are removed at regular intervals during compaction.
9. What are the different types of tombstone markers in HBase for deletion?
There are 3 different types of tombstone markers in HBase for deletion-
1)Family Delete Marker- This markers marks all columns for a column family.
2)Version Delete Marker-This marker marks a single version of a column.
3)Column Delete Marker-This markers marks all the versions of a column.
10. Explain about HLog and WAL in HBase.
All edits in the HStore are stored in the HLog. Every region server has one HLog. HLog contains entries for edits of all regions performed by a particular Region Server.WAL abbreviates to Write Ahead Log (WAL) in which all the HLog edits are written immediately.WAL edits remain in the memory till the flush period in case of deferred log flush.
11. What are the difficulties associated with hotspotting in HBase?
Hotspotting arises when a large amount of client traffic is directed at a single node or a certain few nodes of a cluster. This traffic may be in the form of reads, writes or other database operations. Large amounts of traffic directed at just a few nodes overwhelms the machines responsible for hosting their corresponding regions and results in performance degradation. It could also potentially lead to unavailability of some of the regions as well.
We have further categorized Hadoop HBase Interview Questions for Freshers and Experienced-
Here are few more HBase Interview Questions and Answers
1. Explain about some important Sqoop commands other than import and export.
Create Job (--create)
Here we are creating a job with the name my job, which can import the table data from RDBMS table to HDFS. The following command is used to create a job that is importing data from the employee table in the db database to the HDFS file.
$ Sqoop job --create myjob \
--connect jdbc:mysql://localhost/db \
--username root \
--table employee --m 1
Verify Job (--list)
‘--list’ argument is used to verify the saved jobs. The following command is used to verify the list of saved Sqoop jobs.
$ Sqoop job --list
Inspect Job (--show)
‘--show’ argument is used to inspect or verify particular jobs and their details. The following command and sample output is used to verify a job called myjob.
$ Sqoop job --show myjob
Execute Job (--exec)
‘--exec’ option is used to execute a saved job. The following command is used to execute a saved job called myjob.
$ Sqoop job --exec myjob
2. How Sqoop can be used in a Java program?
The Sqoop jar in classpath should be included in the java code. After this the method Sqoop.runTool () method must be invoked. The necessary parameters should be created to Sqoop programmatically just like for command line.
3. What is the process to perform an incremental data load in Sqoop?
The process to perform incremental data load in Sqoop is to synchronize the modified or updated data (often referred as delta data) from RDBMS to Hadoop. The delta data can be facilitated through the incremental load command in Sqoop.
Incremental load can be performed by using Sqoop import command or by loading the data into hive without overwriting it. The different attributes that need to be specified during incremental load in Sqoop are-
1)Mode (incremental) –The mode defines how Sqoop will determine what the new rows are. The mode can have value as Append or Last Modified.
2)Col (Check-column) –This attribute specifies the column that should be examined to find out the rows to be imported.
3)Value (last-value) –This denotes the maximum value of the check column from the previous import operation.
4. Is it possible to do an incremental import using Sqoop?
Yes, Sqoop supports two types of incremental imports-
To insert only rows Append should be used in import command and for inserting the rows and also updating Last-Modified should be used in the import command.
5. What is the standard location or path for Hadoop Sqoop scripts?
6. How can you check all the tables present in a single database using Sqoop?
The command to check the list of all tables present in a single database using Sqoop is as follows-
Sqoop list-tables –connect jdbc: mysql: //localhost/user;
7. How are large objects handled in Sqoop?
Sqoop provides the capability to store large sized data into a single field based on the type of data. Sqoop supports the ability to store-
1)CLOB ‘s – Character Large Objects
2)BLOB’s –Binary Large Objects
Large objects in Sqoop are handled by importing the large objects into a file referred as “LobFile” i.e. Large Object File. The LobFile has the ability to store records of huge size, thus each record in the LobFile is a large object.
8. Can free form SQL queries be used with Sqoop import command? If yes, then how can they be used?
Sqoop allows us to use free form SQL queries with the import command. The import command should be used with the –e and – query options to execute free form SQL queries. When using the –e and –query options with the import command the –target dir value must be specified.
9. Differentiate between Sqoop and distCP.
DistCP utility can be used to transfer data between clusters whereas Sqoop can be used to transfer data only between Hadoop and RDBMS.
10. What are the limitations of importing RDBMS tables into Hcatalog directly?
There is an option to import RDBMS tables into Hcatalog directly by making use of –hcatalog –database option with the –hcatalog –table but the limitation to it is that there are several arguments like –as-avrofile , -direct, -as-sequencefile, -target-dir , -export-dir are not supported.
11. Is it sugggested to place the data transfer utility sqoop on an edge node ?
It is not suggested to place sqoop on an edge node or gateway node because the high data transfer volumes could risk the ability of hadoop services on the same node to communicate. Messages are the lifeblood of any hadoop service and high latency could result in the whole node being cut off from the hadoop cluster.
We have further categorized Hadoop Sqoop Interview Questions for Freshers and Experienced-
Here are few more frequently asked Sqoop Interview Questions and Answers for Freshers and Experienced
1) Explain about the core components of Flume.
The core components of Flume are –
Event- The single log entry or unit of data that is transported.
Source- This is the component through which data enters Flume workflows.
Sink-It is responsible for transporting data to the desired destination.
Channel- it is the duct between the Sink and Source.
Agent- Any JVM that runs Flume.
Client- The component that transmits event to the source that operates with the agent.
2) Does Flume provide 100% reliability to the data flow?
Yes, Apache Flume provides end to end reliability because of its transactional approach in data flow.
3) How can Flume be used with HBase?
Apache Flume can be used with HBase using one of the two HBase sinks –
Working of the HBaseSink –
In HBaseSink, a Flume Event is converted into HBase Increments or Puts. Serializer implements the HBaseEventSerializer which is then instantiated when the sink starts. For every event, sink calls the initialize method in the serializer which then translates the Flume Event into HBase increments and puts to be sent to HBase cluster.
Working of the AsyncHBaseSink-
AsyncHBaseSink implements the AsyncHBaseEventSerializer. The initialize method is called only once by the sink when it starts. Sink invokes the setEvent method and then makes calls to the getIncrements and getActions methods just similar to HBase sink. When the sink stops, the cleanUp method is called by the serializer.
4) Explain about the different channel types in Flume. Which channel type is faster?
The 3 different built in channel types available in Flume are-
MEMORY Channel – Events are read from the source into memory and passed to the sink.
JDBC Channel – JDBC Channel stores the events in an embedded Derby database.
FILE Channel –File Channel writes the contents to a file on the file system after reading the event from a source. The file is deleted only after the contents are successfully delivered to the sink.
MEMORY Channel is the fastest channel among the three however has the risk of data loss. The channel that you choose completely depends on the nature of the big data application and the value of each event.
5) Which is the reliable channel in Flume to ensure that there is no data loss?
FILE Channel is the most reliable channel among the 3 channels JDBC, FILE and MEMORY.
6) Explain about the replication and multiplexing selectors in Flume.
Channel Selectors are used to handle multiple channels. Based on the Flume header value, an event can be written just to a single channel or to multiple channels. If a channel selector is not specified to the source then by default it is the Replicating selector. Using the replicating selector, the same event is written to all the channels in the source’s channels list. Multiplexing channel selector is used when the application has to send different events to different channels.
7) How multi-hop agent can be setup in Flume?
Avro RPC Bridge mechanism is used to setup Multi-hop agent in Apache Flume.
8) Does Apache Flume provide support for third party plug-ins?
Most of the data analysts use Apache Flume has plug-in based architecture as it can load data from external sources and transfer it to external destinations.
9) Is it possible to leverage real time analysis on the big data collected by Flume directly? If yes, then explain how.
Data from Flume can be extracted, transformed and loaded in real-time into Apache Solr servers using MorphlineSolrSink
10) Differentiate between FileSink and FileRollSink
The major difference between HDFS FileSink and FileRollSink is that HDFS File Sink writes the events into the Hadoop Distributed File System (HDFS) whereas File Roll Sink stores the events into the local file system.
Hadoop Flume Interview Questions and Answers for Freshers - Q.Nos- 1,2,4,5,6,10
Hadoop Flume Interview Questions and Answers for Experienced- Q.Nos- 3,7,8,9
1) Can Apache Kafka be used without Zookeeper?
It is not possible to use Apache Kafka without Zookeeper because if the Zookeeper is down Kafka cannot serve client request.
2) Name a few companies that use Zookeeper.
Yahoo, Solr, Helprace, Neo4j, Rackspace
3) What is the role of Zookeeper in HBase architecture?
In HBase architecture, ZooKeeper is the monitoring server that provides different services like –tracking server failure and network partitions, maintaining the configuration information, establishing communication between the clients and region servers, usability of ephemeral nodes to identify the available servers in the cluster.
4) Explain about ZooKeeper in Kafka
Apache Kafka uses ZooKeeper to be a highly distributed and scalable system. Zookeeper is used by Kafka to store various configurations and use them across the hadoop cluster in a distributed manner. To achieve distributed-ness, configurations are distributed and replicated throughout the leader and follower nodes in the ZooKeeper ensemble. We cannot directly connect to Kafka by bye-passing ZooKeeper because if the ZooKeeper is down it will not be able to serve the client request.
5) Explain how Zookeeper works
ZooKeeper is referred to as the King of Coordination and distributed applications use ZooKeeper to store and facilitate important configuration information updates. ZooKeeper works by coordinating the processes of distributed applications. ZooKeeper is a robust replicated synchronization service with eventual consistency. A set of nodes is known as an ensemble and persisted data is distributed between multiple nodes.
3 or more independent servers collectively form a ZooKeeper cluster and elect a master. One client connects to any of the specific server and migrates if a particular node fails. The ensemble of ZooKeeper nodes is alive till the majority of nods are working. The master node in ZooKeeper is dynamically selected by the consensus within the ensemble so if the master node fails then the role of master node will migrate to another node which is selected dynamically. Writes are linear and reads are concurrent in ZooKeeper.
6) List some examples of Zookeeper use cases.
7) How to use Apache Zookeeper command line interface?
ZooKeeper has a command line client support for interactive use. The command line interface of ZooKeeper is similar to the file and shell system of UNIX. Data in ZooKeeper is stored in a hierarchy of Znodes where each znode can contain data just similar to a file. Each znode can also have children just like directories in the UNIX file system.
Zookeeper-client command is used to launch the command line client. If the initial prompt is hidden by the log messages after entering the command, users can just hit ENTER to view the prompt.
8) What are the different types of Znodes?
There are 2 types of Znodes namely- Ephemeral and Sequential Znodes.
9) What are watches?
Client disconnection might be troublesome problem especially when we need to keep a track on the state of Znodes at regular intervals. ZooKeeper has an event system referred to as watch which can be set on Znode to trigger an event whenever it is removed, altered or any new children are created below it.
10) What problems can be addressed by using Zookeeper?
In the development of distributed systems, creating own protocols for coordinating the hadoop cluster results in failure and frustration for the developers. The architecture of a distributed system can be prone to deadlocks, inconsistency and race conditions. This leads to various difficulties in making the hadoop cluster fast, reliable and scalable. To address all such problems, Apache ZooKeeper can be used as a coordination service to write correct distributed applications without having to reinvent the wheel from the beginning.
Hadoop ZooKeeper Interview Questions and Answers for Freshers - Q.Nos- 1,2,8,9
Hadoop ZooKeeper Interview Questions and Answers for Experienced- Q.Nos-3,4,5,6,7, 10
1) What are different modes of execution in Apache Pig?
Apache Pig runs in 2 modes- one is the “Pig (Local Mode) Command Mode” and the other is the “Hadoop MapReduce (Java) Command Mode”. Local Mode requires access to only a single machine where all files are installed and executed on a local host whereas MapReduce requires accessing the Hadoop cluster.
2) Explain about co-group in Pig.
COGROUP operator in Pig is used to work with multiple tuples. COGROUP operator is applied on statements that contain or involve two or more relations. The COGROUP operator can be applied on up to 127 relations at a time. When using the COGROUP operator on two tables at once-Pig first groups both the tables and after that joins the two tables on the grouped columns.
We have further categorized Hadoop Pig Interview Questions for Freshers and Experienced-
Here are a few more frequently asked Pig Hadoop Interview Questions and Answers for Freshers and Experienced
1) Explain about the SMB Join in Hive.
In SMB join in Hive, each mapper reads a bucket from the first table and the corresponding bucket from the second table and then a merge sort join is performed. Sort Merge Bucket (SMB) join in hive is mainly used as there is no limit on file or partition or table join. SMB join can best be used when the tables are large. In SMB join the columns are bucketed and sorted using the join columns. All tables should have the same number of buckets in SMB join.
2) How can you connect an application, if you run Hive as a server?
When running Hive as a server, the application can be connected in one of the 3 ways-
ODBC Driver-This supports the ODBC protocol
JDBC Driver- This supports the JDBC protocol
Thrift Client- This client can be used to make calls to all hive commands using different programming language like PHP, Python, Java, C++ and Ruby.
3) What does the overwrite keyword denote in Hive load statement?
Overwrite keyword in Hive load statement deletes the contents of the target table and replaces them with the files referred by the file path i.e. the files that are referred by the file path will be added to the table when using the overwrite keyword.
4) What is SerDe in Hive? How can you write your own custom SerDe?
SerDe is a Serializer DeSerializer. Hive uses SerDe to read and write data from tables. Generally, users prefer to write a Deserializer instead of a SerDe as they want to read their own data format rather than writing to it. If the SerDe supports DDL i.e. basically SerDe with parameterized columns and different column types, the users can implement a Protocol based DynamicSerDe rather than writing the SerDe from scratch.
5) What are the differences between internal and external tables in Hive?
Data tables created inside the database are considered internal tables of Hive by default.
External tables are found on shareable data locations in the HDFS outside the database
Hive has ownership of the internal tables and manages the metadata and the data within internal tables by itself.
Hive does not have ownership of external tables and only manages the metadata associated with the tables.
Both metadata and the table data will be removed when the table is dropped.
Only metadata is removed on dropping the table.
Internal tables support the TRUNCATE command.
External tables do not support the TRUNCATE command.
Internal tables support query result caching (the results of a particular query are temporarily stored if needed for use in a subsequent query).
External tables do not support query result caching.
6) Consider the following two queries. What is the difference in the query results?
SELECT x.value, y.value FROM x LEFT OUTER JOIN y ON (x.key=y.key)
SELECT x.value, y.value FROM x RIGHT OUTER JOIN y ON (x.key=y.key)
In both Queries 1 and 2, the output rows will contain a list of x.value and y.value for all cases where the values of x.key and y.key are equal.
The difference here is that, in query 1, if there is no corresponding value ‘value’ in y for x.value it will be added to the output query as ‘x.value,NULL’, since x is to the left of y (x LEFT OUTER JOIN y). All values for y.value which do not have a corresponding x.value will be dropped from the query result of query 1.
In the case of query 2, it is vice versa, since it is a right outer join and on the right, there is y. This means that in the case of Query 2, all y values which do not have a corresponding x value will be present in the result query as ‘NULL, y.value’ but all x values without a corresponding y value will be dropped.
7) How can one list down all partitions of a table in Hive?
There are two ways to list all the partitions of a table in Hive.
Using Hive prompt
$hive -e show partitions table_name;
Using Hive command
$ hive> show partitions table_name;
We have further categorized Hadoop Hive Interview Questions for Freshers and Experienced-
Hadoop Hive Interview Questions and Answers for Freshers- Q.Nos-3,5,7
Hadoop Hive Interview Questions and Answers for Experienced- Q.Nos-1,2,4,6
Here are a few more frequently asked Hadoop Hive Interview Questions and Answers for Freshers and Experienced
1)What are the stable versions of Hadoop?
Release 2.7.1 (stable)
Release 1.2.1 (stable)
2) What is Apache Hadoop YARN?
YARN is a powerful and efficient feature rolled out as a part of Hadoop 2.0.YARN is a large scale distributed system for running big data applications.
3) Is YARN a replacement of Hadoop MapReduce?
YARN is not a replacement of Hadoop but it is a more powerful and efficient technology that supports MapReduce and is also referred to as Hadoop 2.0 or MapReduce 2.
4) What are the additional benefits YARN brings in to Hadoop?
5) How can native libraries be included in YARN jobs?
There are two ways to include native libraries in YARN jobs-
1) By setting the -Djava.library.path on the command line but in this case there are chances that the native libraries might not be loaded correctly and there is possibility of errors.
2) The better option to include native libraries is to set the LD_LIBRARY_PATH in the .bashrc file.
6) Explain the differences between Hadoop 1.x and Hadoop 2.x
7) What are the core changes in Hadoop 2.0?
Hadoop 2.x provides an upgrade to Hadoop 1.x in terms of resource management, scheduling and the manner in which execution occurs. In Hadoop 2.x the cluster resource management capabilities work in isolation from the MapReduce specific programming logic. This helps Hadoop to share resources dynamically between multiple parallel processing frameworks like Impala and the core MapReduce component. Hadoop 2.x Hadoop 2.x allows workable and fine grained resource configuration leading to efficient and better cluster utilization so that the application can scale to process larger number of jobs.
8) Differentiate between NFS, Hadoop NameNode and JournalNode.
HDFS is a write once file system so a user cannot update the files once they exist either they can read or write to it. However, under certain scenarios in the enterprise environment like file uploading, file downloading, file browsing or data streaming –it is not possible to achieve all this using the standard HDFS. This is where a distributed file system protocol Network File System (NFS) is used. NFS allows access to files on remote machines just similar to how local file system is accessed by applications.
Namenode is the heart of the HDFS file system that maintains the metadata and tracks where the file data is kept across the Hadoop cluster.
StandBy Nodes and Active Nodes communicate with a group of light weight nodes to keep their state synchronized. These are known as Journal Nodes.
9) What are the modules that constitute the Apache Hadoop 2.0 framework?
Hadoop 2.0 contains four important modules of which 3 are inherited from Hadoop 1.0 and a new module YARN is added to it.
CLICK HERE to read more about the YARN module in Hadoop 2.x.
10) How is the distance between two nodes defined in Hadoop?
Measuring bandwidth is difficult in Hadoop so network is denoted as a tree in Hadoop. The distance between two nodes in the tree plays a vital role in forming a Hadoop cluster and is defined by the network topology and java interface DNStoSwitchMapping. The distance is equal to the sum of the distance to the closest common ancestor of both the nodes. The method getDistance(Node node1, Node node2) is used to calculate the distance between two nodes with the assumption that the distance from a node to its parent node is always 1.
We have further categorized Hadoop YARN Interview Questions for Freshers and Experienced-
11) What are the key components of YARN?
Hadoop YARN consists of the following components:
Resource Manager: The Resource manager handles all the allocation of resources in the cluster and management of applications that have to run. The Resource manager is the ultimate authority for resource allocation. When it receives a request for the processing of an application, it forwards this request to the node manager concerned.
Node Manager: The Node manager is responsible for handling an individual node in a Hadoop cluster. It manages tasks and applications on the node that it is responsible for handling. The Node manager is registered with the resource manager. It informs the resource manager about the health status of the node by sending heartbeat messages. The primary goal of the node manager is to manage application containers that are assigned to it by the resource manager. It also keeps track of the resource usage of the containers in the node.
Application Master: In Hadoop, an application refers to a single job that is submitted to the framework. Each application has a unique Application Master associated with it, that is specific to that particular framework. The Application Master coordinates the execution of an application in the cluster and manages any faults which may occur. It negotiates resources and containers based on input from the resource manager. Periodic heartbeats are sent to the Resource Manager to keep it updated with respect to the health and resource demands associated with the resources. The Application Master along with the Node Manager monitor and execute the tasks carried out by various components.
Container: The container is a collection of physical resources including RAM, CPU cores and disks present in a single node of a cluster. Containers in YARN are managed by the CLC (container life-cycle). The CLC contains details about the environment variables, various dependencies stored in remotely accessible storage, security tokens and the payload associated with Node Manager services. The container grants an application the rights to use a specific amount of resources from a particular host.
12) What do you know about the resource manager available in YARN?
The Resource Manager is the ultimate authority and the master daemon of YARN. it is responsible for allocation of resources and management of all the applications. Whenever the resource manager receives a processing request, it forwards this request to the corresponding node manager and allocates resources for the execution of the request. There are two major components:
Scheduler: it is responsible for performing scheduling of tasks based on the applications allotted and availability of resources. The Scheduler does not perform any other tasks beyond scheduling, such as monitoring or tracking of resources. It does not guarantee restarting of tasks in cases of failure due to either hardware or software failures.
Application manager: the application manager is responsible for accepting applications and negotiating for a container to execute the ApplicationMaster which is specific to applications. It also provides the service for restarting the ApplicationMaster in case of any failures. The Application manager maintains a collection of applications that have been submitted. It also maintains a cache of applications that have been completed.
13) What are the scheduling policies available in Apache YARN?
There are three types of scheduling policies that are provided by YARN. These are:
FIFO scheduler: In the First in first out (FIFO) scheduler, the tasks or applications that come first will be executed first. The tasks are placed into the queue when submitted and are serviced in the order in which they enter the queue. Once a job begins getting executed, no intervention is allowed from other jobs. This is the default scheduling policy that is used in Hadoop, and hence, does not require any special configuration. The downside to this scheduler is that even if a particular task is of high priority, it has no choice but to wait its turn to be executed.
Fair Scheduler: The Fair Scheduler works a little similar to the Capacity Scheduler, however, the priority of jobs is taken into account. By means of the Fair Scheduler, YARN applications can distribute the resources in large Hadoop clusters in such a way that all applications within the cluster are handled in an equal amount of time. The Fair Scheduler takes decisions based on memory availability and maintains resources dynamically. Whenever a high priority task arrives in a queue, the task is executed in parallel by replacing some portion of the slots which are already dedicated. The Fair Scheduler is a good option when it comes to the management of high priority tasks.
1) How will you test data quality?
The entire data that has been collected could be important but all data is not equal so it is necessary to first define from where the data came, how the data would be used and consumed. Data that will be consumed by vendors or customers within the business ecosystem should be checked for quality and needs to be cleaned. This can be done by applying stringent data quality rules and by inspecting different properties like conformity, perfection, repetition, reliability, validity, completeness of data, etc. The collected data might have issues like NULL values, outliers, data type issues, encoding issues, language issues, column shift issues, special characters, header issues etc. So it is important to have a data cleaning and validation framework in place to clean and validate the data issues to ensure data completeness. We can use python to read the incoming data into pandas data frames and perform various checks on the data and transform the data into the required format, clean the data, validate the data, and store it into the data lake or HDFS for further processing
2) What are the challenges that you encounter when testing large datasets?
3) What do you understand by MapReduce validation ?
This the subsequent and most important step of the big data testing process. Hadoop developer needs to verify the right implementation of the business logic on every hadoop cluster node and validate the data after executing it on all the nodes to determine -
For many hadoop job seekers, the question from the interviewer – “Do you have any questions for me?” indicates the end of a Hadoop developer job interview. It is always enticing for a Hadoop job seeker to immediately say “No” to the question for the sake of keeping the first impression intact.However, to land a hadoop job or any other job, it is always preferable to fight that urge and ask relevant questions to the interviewer.
Asking questions related to the Hadoop technology implementation, shows your interest in the open hadoop job role and also conveys your interest in working with the company.Just like any other interview, even hadoop interviews are a two-way street- it helps the interviewer decide whether you have the desired hadoop skills they in are looking for in a hadoop developer, and helps an interviewee decide if that is the kind of big data infrastructure and hadoop technology implementation you want to devote your skills for foreseeable future growth in the big data domain.
Candidates should not be afraid to ask questions to the interviewer. To ease this for hadoop job seekers, ProjectPro has collated few hadoop interview FAQ’s that every candidate should ask an interviewer during their next hadoop job interview-
1) What is the size of the biggest hadoop cluster a company X operates?
Asking this question helps a hadoop job seeker understand the hadoop maturity curve at a company.Based on the answer of the interviewer, a candidate can judge how much an organization invests in Hadoop and their enthusiasm to buy big data products from various vendors. The candidate can also get an idea on the hiring needs of the company based on their hadoop infrastructure.
2) For what kind of big data problems, did the organization choose to use Hadoop?
Asking this question to the interviewer shows the candidates keen interest in understanding the reason for hadoop implementation from a business perspective. This question gives the impression to the interviewer that the candidate is not merely interested in the hadoop developer job role but is also interested in the growth of the company.
3) Based on the answer to question no 1, the candidate can ask the interviewer why the hadoop infrastructure is configured in that particular way, why the company chose to use the selected big data tools and how workloads are constructed in the hadoop environment.
Asking this question to the interviewer gives the impression that you are not just interested in maintaining the big data system and developing products around it but are also seriously thoughtful on how the infrastructure can be improved to help business growth and make cost savings.
4) What kind of data the organization works with or what are the HDFS file formats the company uses?
The question gives the candidate an idea on the kind of big data he or she will be handling if selected for the hadoop developer job role. Based on the data, it gives an idea on the kind of analysis they will be required to perform on the data.
5) What is the most complex problem the company is trying to solve using Apache Hadoop?
Asking this question helps the candidate know more about the upcoming projects he or she might have to work and what are the challenges around it. Knowing this beforehand helps the interviewee prepare on his or her areas of weakness.
6) Will I get an opportunity to attend big data conferences? Or will the organization incur any costs involved in taking advanced hadoop or big data certification?
This is a very important question that you should be asking these the interviewer. This helps a candidate understand whether the prospective hiring manager is interested and supportive when it comes to professional development of the employee.
So, you have cleared the technical interview after preparing thoroughly with the help of the Hadoop Interview Questions shared by ProjectPro. After an in-depth technical interview, the interviewer might still not be satisfied and would like to test your practical experience in navigating and analysing big data. The expectation of the interviewer is to judge whether you are really interested in the open position and ready to work with the company, regardless of the technical knowledge you have on hadoop technology.
There are quite a few on-going debates in the hadoop community, on the advantages of the various components in the hadoop ecosystem-- for example what is better MapReduce, Pig or Hive or Spark vs. Hadoop or when should a company use MapReduce over other alternative? etc. Interviewee and Interviewer should both be ready to answer such hadoop interview FAQs, as there is no right or wrong answer to these questions.The best possible way to answer these Hadoop interview FAQs is to explain why a particular interviewee favours an option. Answering these hadoop interview FAQs with practical examples as to why the candidate favours an option, demonstrates his or her understanding of the business needs and helps the interviewer judge the flexibility of the candidate to use various big data tools in the hadoop ecosystem.
Here are a few hadoop interview FAQs that are likely to be asked by the interviewer-
1) If you are an experienced hadoop professional then you are likely to be asked questions like –
Your answer to these interview questions will help the interviewer understand your expertise in Hadoop based on the size of the hadoop cluster and number of nodes. Based on the highest volume of data you have handled in your previous projects, interviewer can assess your overall experience in debugging and troubleshooting issues involving huge hadoop clusters.
The number of tools you have worked with help an interviewer judge that you are aware of the overall hadoop ecosystem and not just MapReduce. To be selected, it all depends on how well you communicate the answers to all these questions.
2) What are the challenges that you faced when implementing hadoop projects?
Interviewers are interested to know more about the various issues you have encountered in the past when working with hadoop clusters and understand how you addressed them. The way you answer this question tells a lot about your expertise in troubleshooting and debugging hadoop clusters.The more issues you have encountered, the more probability there is, that you have become an expert in that area of Hadoop. Ensure that you list out all the issues that have trouble-shooted.
3) How were you involved in data modelling, data ingestion, data transformation and data aggregation?
You are likely to be involved in one or more phases when working with big data in a hadoop environment. The answer to this question helps the interviewer understand what kind of tools you are familiar with.If you answer that your focus was mainly on data ingestion then they can expect you to be well-versed with Sqoop and Flume, if you answer that you were involved in data analysis and data transformation then it gives the interviewer an impression that you have expertise in using Pig and Hive.
4) What is your favourite tool in the hadoop ecosystem?
The answer to this question will help the interviewer know more about the big data tools that you are well-versed with and are interested in working with. If you show affinity towards a particular tool then the probability that you will be deployed to work on that particular tool, is more.If you say that you have a good knowledge of all the popular big data tools like pig, hive, HBase, Sqoop, flume then it shows that you have knowledge about the hadoop ecosystem as a whole.
5) In you previous project, did you maintain the hadoop cluster in-house or used hadoop in the cloud?
Most of the organizations still do not have the budget to maintain Hadoop cluster in-house and they make use of Hadoop in the cloud from various vendors like Amazon, Microsoft, Google, etc. Interviewer gets to know about your familiarity with using hadoop in the cloud because if the company does not have an in-house implementation then hiring a candidate who has knowledge about using hadoop in the cloud is worth it.
1) Write a MapReduce program to add all the elements in a file. (Hadoop Developer Interview Question asked at KPIT)
2) What is the difference between HashSet and hashmap? (Big Data Interview Question asked at Wipro)
3) Write a Hive program to find the number of employees department-wise in an organization. ( Hadoop Developer Interview Question asked at Tripod Technologies)
4) How will you read a CSV file of 10GB and store it in the database as it is in just few seconds? (Hadoop Interview Question asked at Deutsche Bank)
5) How will a file of 100MB be stored in Hadoop? (Hadoop Interview Question asked at Deutsche Bank)
6) Given the HTTP URL, write a regular expression to extract different parts of the URL like server, path, and protocol.
7) If you store a file from HDFS to Apache Pig using PIGSTORAGE in grunt shell but if that file is not really available in that HDFS path will the command work?
8) Apart from partitioning and bucketing, what are the other methods you can use for improving performance?
9) Which file format is better Parquet or ORC?
Big Data and Hadoop Job Growth Trends
Big Data and Hadoop is a constantly changing field that required people to quickly upgrade their skills, to fit the requirements for Hadoop related jobs. If you are applying for a Hadoop job role, it is best to be prepared to answer any Hadoop interview question that might come your way. We will keep updating this list of Hadoop Interview questions, to suit the current industry standards.
With more than 30,000 open Hadoop developer jobs, professionals must familiarize themselves with each and every component of the Hadoop ecosystem to make sure that they have a deep understanding of what Hadoop is so that they can form an effective approach to a given big data problem. To help you get started, ProjectPro presented a comprehensive list of Top 50 Hadoop Developer Interview Questions asked during recent Hadoop job interviews. In case you are appearing for a Hadoop administrator interview, we've got you covered with your hadoop admin job interview preparation, check out these top Hadoop admin interview questions and answers.
We had to spend lots of hours researching and deliberating on what are the best possible answers to these interview questions. We would love to invite people from the industry – hadoop developers, hadoop admins and architects to kindly help us and everyone else – with answering the unanswered questions if any.
Did you find these hadoop interview FAQ’s useful and think that one of the prospective hadoopers will benefit from it? If yes, then please use the social media share buttons to help the big data community at large.