Predicting Flight Delays using Apache Spark and Kylin

Predicting Flight Delays using Apache Spark and Kylin

In this project, we will be building and querying an OLAP Cube for Flight Delays on the Hadoop platform.

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

Customer Love

Read All Reviews

SUBHABRATA BISWAS

Lead Consultant, ITC Infotech

The project orientation is very much unique and it helps to understand the real time scenarios most of the industries are dealing with. And there is no limit, one can go through as many projects... Read More

Ray Han

Tech Leader | Stanford / Yale University

I think that they are fantastic. I attended Yale and Stanford and have worked at Honeywell,Oracle, and Arthur Andersen(Accenture) in the US. I have taken Big Data and Hadoop,NoSQL, Spark, Hadoop... Read More

What will you learn

Discuss the installation of Apache Kylin in a Hortonworks sandbox.
Design star schema on our flight dataset
Implementing our star schema in Kylin
Building and merging Kyline segments incrementally.
Building Cubes using Kylin Restful API
How to execute Cubes using Spark Engine

Project Description

In previous Hackerday sessions, we have introduced how to bring OLAP to extremely large datasets in Apache Kylin. For those who don't know what Kylin is, Kylin (kylin.apache.org) is a Distributed Analytics Engine that provides SQL interface and multidimensional analysis (OLAP) on the large dataset using MapReduce or Spark. This means that I can answer classical aggregate queries in the Hadoop platform with a low latency over billions of records.

In this Hackerday, we will be performing an OLAP cube design using the flight on-time dataset. Since we have previously introduced Kylin, this Hackerday session will look at more involved features like incremental build, performance tuning or consideration tips, we will discuss the Spark engine as well as how to build different types of model.

Similar Projects

This Elasticsearch example deploys the AWS ELK stack to analyse streaming event data. Tools used include Nifi, PySpark, Elasticsearch, Logstash and Kibana for visualisation.

The goal of this spark project for students is to explore the features of Spark SQL in practice on the latest version of Spark i.e. Spark 2.0.

PySpark Project-Get a handle on using Python with Spark through this hands-on data processing spark python tutorial.

Curriculum For This Mini Project