Building Real-Time Data Pipelines with Kafka Connect

Building Real-Time Data Pipelines with Kafka Connect

In this big data project, we will see how data ingestion and loading is done with Kafka connect APIs while transformation will be done with Kafka Streaming API.
explanation image

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

ipython image

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

project experience

Project Experience

Add project experience to your Linkedin/Github profiles.

Customer Love

Read All Reviews
profile image

SUBHABRATA BISWAS linkedin profile url

Lead Consultant, ITC Infotech

The project orientation is very much unique and it helps to understand the real time scenarios most of the industries are dealing with. And there is no limit, one can go through as many projects... Read More

profile image

Hiren Ahir linkedin profile url

Microsoft Azure SQL Sever Developer, BI Developer

I'm a Graduate student and came into the job market and found a university degree wasn't sufficient to get a good paying job. I aimed at hottest technology in the market Big Data but the word BigData... Read More

What will you learn

Kafka and Data warehousing
Real-time data warehousing
Kafka Connect API
Kafka Streams API
End-to-end Kafka pipeline

Project Description

Lately, the phrase "ETL is dead" has become more popular. But that statement is flatly false. It should rather have been "Batch ETL is growing unpopular". Companies now believe not only in the power of data but also in the power of current-ness of data. This means that a dashboard that reveals sales pattern for yesterday is less correct than one that shows sales pattern in the last 30 minutes.

Kafka is a scalable and distributed streaming and messaging platform is a great choice for building today's ETL pipeline.

In this big data kafka project, we will see this in theory as well as implementation. We will see how data ingestion and loading is done with Kafka connect APIs while transformation will be done with Kafka Streaming API. But this is not all.

Similar Projects

Work with Streaming Data using Twitter API to Build a JobPortal

In this spark streaming project, we are going to build the backend of a IT job ad website by streaming data from twitter for analysis in spark.

View Project Details

Build a big data pipeline with AWS Quicksight, Druid, and Hive

Use the dataset on aviation for analytics to simulate a complex real-world big data pipeline based on messaging with AWS Quicksight, Druid, NiFi, Kafka, and Hive.

View Project Details

Web Server Log Processing using Hadoop

In this hadoop project, you will be using a sample application log file from an application server to a demonstrated scaled-down server log processing pipeline.

View Project Details

Curriculum For This Mini Project

1-Dec-2017
02h 43m
8-Dec-2017
03h 29m