1-844-696-6465 (US)        +91 77600 44484        help@dezyre.com

Work with Streaming Data using Twitter API to Build a JobPortal

In this spark streaming project, we are going to build the backend of a IT job ad website by streaming data from twitter for analysis in spark.

Users who bought this project also bought

What will you learn

  • Streaming twitter using flume
  • Integrating flume with Spark-Streaming for processing twitter events
  • Data processing with Spark
  • Integrating Kafka to complex event alert
  • Integrating spark with online databases
  • Coordinating the data processing pipeline with Oozie

What will you get

  • Access to recording of the complete project
  • Access to all material related to project like data files, solution files etc.


  • It is expected that students have a fair knowledge of Big Data and Hadoop.
  • This project requires a lot of coding so there will very little time to go over the concepts for each piece of tech to be used.
  • Installation of the Cloudera quickstart vm is super-essential to get the best from this class.
  • Instruction on how to setup a scala SDK and runtime can be found from here

Project Description

In this spark project, we are going to be building a business. Yes, a business that is similar to a IT job ad site. This Job portal will stream data from twitter to locate recently published IT jobs, process them and make them available via a simple search api. Also, to complete the circle, we will be building notification features to user who subscribe for job ads notification.

On completion of this big data project, we will provide a job portal for every IT job tweeted and provide an apply-early advantage to users.



Big Data & Enterprise Software Engineer

I am passionate about software development, databases, data analysis and the android platform. My native language is java but no one has stopped me so far from learning and using angular and node.js. Data and data analysis is thrilling and so are my experiences with SQL on Oracle, Microsoft SQL Server, Postgres and MyS see more...

Curriculum For This Mini Project

02h 33m
02h 34m