1-844-696-6465 (US)        +91 77600 44484        help@dezyre.com
real-time-auto-tracking-with-spark-redis.jpg

Real-time Auto Tracking with Spark-Redis

Spark Project - Discuss real-time monitoring of taxis in a city. The real-time data streaming will be simulated using Flume. The ingestion will be done using Spark Streaming.

Users who bought this project also bought

What will you learn

  • Key-Value (NoSQL) Databases
  • Using Redis as a pub/sub message-oriented middleware
  • Using Redis as a caching server/persistence store
  • Streaming data with Flume/Spark integration
  • Real-time processing and display of streamed data on a "dashboard".
  • Compare the capability of Redis as a pub/sub middleware with that of the industry appraised Apache Kafka.
  • How to extends the appllication to production grade.

What will you get

  • Access to recording of the complete project
  • Access to all material related to project like data files, solution files etc.

Prerequisites

  • Download the latest release of Redis
  • The installation will be handled in class

Project Description

The era of IOT brought with it the need to stream data, process and sometimes display its information in real or near-real time. 
In this spark streaming project, we will be using a dataset that passes for real-time data sensor feeds for tracking auto vehicles around the city of Bejing. We will track each vehicle as the signal is received from our streaming simulation (using Flume). We will receive the streams of data using Spark Streaming and use the Redis as a pub/sub middleware.

Furthermore, we will use a java swing based application to display real-time information about all vehicles being tracked. While tracking the vehicle, we will be looking for indexes like current speed, total time and distance covered.

While this spark project is about tracking autos, the principles shared in this big data project will cover wide areas of implementing real-time sensor data processing and much more IOT.

Instructors

 
Michael

Big Data & Enterprise Software Engineer

I am passionate about software development, databases, data analysis and the android platform. My native language is java but no one has stopped me so far from learning and using angular and node.js. Data and data analysis is thrilling and so are my experiences with SQL on Oracle, Microsoft SQL Server, Postgres and MyS see more...

Curriculum For This Mini Project

 
  Discussion on Project Requirements
04m
  Walkthrough of the Application
01m
  Start Cloudera VM and Put Application on Eclipse
01m
  Agenda for the Session
03m
  Install and Start Redis Server on Quickstart VM
02m
  Introduction to NoSQL Systems
02m
  Integration of Application at Database Level
08m
  What is Redis?
04m
  Datatypes in Redis
01m
  Use Cases for Redis
06m
  Exploring and Working with Redis Commands
14m
  How to Use Redis as a Persistent Store?
01m
  What are Messaging Servers?
02m
  Redis - Publish-Subscribe
08m
  Commands to use Redis for Publish Subscribe
02m
  Redis Drivers -Jedis
01m
  Open Quickstart VM and Fire Up Eclipse
02m
  Introduction to T-Drive Trajectory Dataset
02m
  Overview of the Application -Project in a Nutshell
10m
  Use of MongoDB to extend the application to production grade
05m
  Recap of the Previous Session
05m
  Agenda for the Session
02m
  Real-Time Analytics Use Cases on Streaming Data
08m
  Spark Streaming -Real-Time and Near Real-Time Streaming
05m
  Flume for Real-Time Streaming
05m
  Spark Streaming Concepts
04m
  Flume Integration
01m
  Using Avro Sink for Flume Integration
05m
  Redis Performance Benchmarks
02m
  Processing logic to Initiate Streaming and Streaming Content
04m
  Spark Streaming Execution
04m
  Flume Agent Connection to Listen to Data (Run the Flume Agents)
01m
  Dashboard Visualization of Taxis in Real-Time
08m
  Sequence of Commands for Execution
01m
  Debugging
02m
  Making Redis Faster with Buffer as ConcurrentLinkedQueue
05m
  Redis vs Kafka
06m
  Processing Completed
01m