1-844-696-6465 (US)        +91 77600 44484        help@dezyre.com
airline-online-performance.jpg

Airline Online Performance

In this project, we are going to make big data available and accessible.
4.84.8

Users who bought this project also bought

What will you learn

  • Data preprocessing with Pig
  • Hive vs. MPP database systems (Hive vs. Impala/Drill)
  • Hive/Impala partitioning and clustering
  • Data compression, tuning and query optimization
  • Using database views to represent data.
  • Building time series data model
  • Visuliazing data using Microsoft Excel via ODBC

What will you get

  • Access to recording of the complete project
  • Access to all material related to project like data files, solution files etc.

Prerequisites

  • For purpose of visualization, it is expected that you have Microsoft Excel on your host machine or an equivalent.

Project Description

Before data on any platform will become an asset to any organization, it has to pass through processing stage to ensure quality and availability. Afterward, that data has to be available to users (both human and system users). The availability of quality data in any organization is the guarantee of the value that data science (in general) will be to that organization. 

We are using the airline on-time performance dataset to demonstrate these principles and techniques in this hackerday and we will proceed to answer questions that can be found on the website like:

  • When is the best time of day/day of week/time of year to fly to minimize delays?
  • Do older planes suffer more delays?
  • How does the number of people flying between different locations change over time?

We will also transform the data access model into time series and demonstrate how clients can access data in our big data infrastructure using a simple tool like the Excel spreadsheet.

Instructors

 
Michael

Big Data & Enterprise Software Engineer

I am passionate about software development, databases, data analysis and the android platform. My native language is java but no one has stopped me so far from learning and using angular and node.js. Data and data analysis is thrilling and so are my experiences with SQL on Oracle, Microsoft SQL Server, Postgres and MyS see more...

Curriculum For This Mini Project

 
  Introduction to Data Infrastructure
00:07:55
  Methods to ingest data in a data infrastructure
00:06:51
  Messaging Layer Example
00:11:10
  Small File Problem
00:03:59
  Business problem overview and topics covered
00:02:29
  Hive JDBC and Impala ODBC drivers
00:02:21
  Data Pre-processing
00:06:10
  Data Extraction and Loading
00:03:06
  Setting up the Datawarehouse
00:13:55
  Creating Data Table
00:02:26
  Impala Architecture
00:14:11
  Working with Hive versus Impala & File Formats
00:08:36
  Hive query for Airline data analysis + Parquet - 1
00:21:10
  Hive query for Airline data analysis + Parquet - 2
00:05:53
  Hive query for Airline data analysis + Parquet - 3
00:16:41
  Read and write data to tables
00:16:34
  Parquet data compression
00:06:46
  Calculate average flight delay
00:10:11
  Partitioning Basics
00:02:55
  Where to do the data processing - Hive or Impala ?
00:10:36
  Partitioning Calculations
00:15:59
  Dynamic Paritioninig
00:04:02
  Clustering, Sampling, Bucketed Tables
00:13:39
  Hive Compression and Execution Engine
00:15:52
  Impala COMPUTE STATS and File Formats
00:13:57
  Using database views to represent data
00:15:33