Airline Dataset Analysis using Hadoop, Hive, Pig and Impala

Airline Dataset Analysis using Hadoop, Hive, Pig and Impala

Hadoop Project- Perform basic big data analysis on airline dataset using big data tools -Pig, Hive and Impala.


Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

Customer Love

Read All Reviews

Swati Patra

Systems Advisor , IBM

I have 11 years of experience and work with IBM. My domain is Travel, Hospitality and Banking - both sectors process lots of data. The way the projects were set up and the mentors' explanation was... Read More

Camille St. Omer

Artificial Intelligence Researcher, Quora 'Most Viewed Writer in 'Data Mining'

I came to the platform with no experience and now I am knowledgeable in Machine Learning with Python. No easy thing I must say, the sessions are challenging and go to the depths. I looked at graduate... Read More

What will you learn

Introduction to Data infrastructures Methods for ingestion of data(Backend Service, Data Warehouse)
Tackling Small file problem
Roadmap of the project and business problem
Hive JDBC and Impala ODBC driver
Extracting and loading the data in Cloudera VMware
Data preprocessing with Pig
Writing Queries in Hue Hive for creating tables
Hive vs. MPP database systems (Hive vs. Impala/Drill)
Basic EDA using Hive
Hive/Impala partitioning and clustering
Writing data from Pig to Hive directly using HCatloader
Data compression, tuning and query optimization using parquet
Using database views to represent data
Clustering , Sampling and Bucketed Tables
Building time series data model
Impala compute Stats and File format
Visualizing data using Microsoft Excel via ODBC

Project Description

Before data on any platform will become an asset to any organization, it has to pass through processing stage to ensure quality and availability. Afterward, that data has to be available to users (both human and system users). The availability of quality data in any organization is the guarantee of the value that data science (in general) will be to that organization. 

We are using the airline on-time performance dataset (flights data csv) to demonstrate these principles and techniques in this hadoop project and we will proceed to answer the below questions -

  • When is the best time of day/day of week/time of year to fly to minimize delays?
  • Do older planes suffer more delays?
  • How does the number of people flying between different locations change over time?

We will also transform the data access model into time series and demonstrate how clients can access data in our big data infrastructure using a simple tool like the Excel spreadsheet.

Similar Projects

Learn to write a Hadoop Hive Program for real-time querying.

In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security

In this hadoop project, learn about the features in Hive that allow us to perform analytical queries over large datasets.

Curriculum For This Mini Project

Introduction to Data Infrastructure
Methods to ingest data in a data infrastructure
Messaging Layer Example
Small File Problem
Business problem overview and topics covered
Hive JDBC and Impala ODBC drivers
Data Pre-processing
Data Extraction and Loading
Setting up the Datawarehouse
Creating Data Table
Impala Architecture
Working with Hive versus Impala & File Formats
Hive query for Airline data analysis + Parquet - 1
Hive query for Airline data analysis + Parquet - 2
Hive query for Airline data analysis + Parquet - 3
Read and write data to tables
Parquet data compression
Calculate average flight delay
Partitioning Basics
Where to do the data processing - Hive or Impala ?
Partitioning Calculations
Dynamic Paritioninig
Clustering, Sampling, Bucketed Tables
Hive Compression and Execution Engine
Impala COMPUTE STATS and File Formats
Using database views to represent data
Using Excel or Qlikview for Visualization