In this hadoop project, we are going to be continuing the series on data engineering by discussing and implementing various ways to solve the hadoop small file problem.
What is Small file problem in Hadoop
How it arises (Batch and Streaming mode)
Solution (Streaming): Using flume
Solution (Streaming): Preprocessing and storing in a NoSQL database
Solution (Batch): Merging before storing in HDFS
and more...
Online Hadoop Projects -Solving small file problem in Hadoop
In this hadoop project, we are going to be continuing the series on data engineering by discussing and implementing various ways to solve the hadoop small file problem.
In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security
Making a case for real time processing of log files
Getting logs at real time using Flume Log4J appenders
Making a case for Kafka for log aggregation.
Storing log event as a time series datasets in HBase
Integrating Hive and HBase for data retrieval using query.
and more...
Real-Time Log Processing using Spark Streaming Architecture
In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security
In this big data project, we will look at how to mine and make sense of connections in a simple way by building a Spark GraphX Algorithm and a Network Crawler.
Designing your Github network of persons
Building the network model in HBase
Running your network crawler
Using spark to analyze the network
Running graph algorithms using GraphFrame or Spark GraphX
Design a Network Crawler by Mining Github Social Profiles
In this big data project, we will look at how to mine and make sense of connections in a simple way by building a Spark GraphX Algorithm and a Network Crawler.
In this big data project, we will be performing an OLAP cube design using AdventureWorks database. The deliverable for this session will be to design a cube, build and implement it using Kylin, query the cube and even connect familiar tools (like Excel) with our new cube.
Apache Kylin and how it works?
Installing Apache Kylin in our Quickstart VM
Design star schema on our AdventureWorks database
Implementing our star schema in Kylin
Writing aggregate queries against a Kylin cube
and more...
Implementing OLAP on Hadoop using Apache Kylin
In this big data project, we will be performing an OLAP cube design using AdventureWorks database. The deliverable for this session will be to design a cube, build and implement it using Kylin, query the cube and even connect familiar tools (like Excel) with our new cube.
In this hadoop project, we are going to be continuing the series on data engineering by discussing and implementing various ways to solve the hadoop small file problem.
What is Small file problem in Hadoop
How it arises (Batch and Streaming mode)
Solution (Streaming): Using flume
Solution (Streaming): Preprocessing and storing in a NoSQL database
Solution (Batch): Merging before storing in HDFS
and more...
Online Hadoop Projects -Solving small file problem in Hadoop
In this hadoop project, we are going to be continuing the series on data engineering by discussing and implementing various ways to solve the hadoop small file problem.
In this hadoop project, you will be using a sample application log file from an application server to a demonstrated scaled-down server log processing pipeline.
The benefits of log-mining in certain industries
A full log-mining application use-case
Using Flume to ingest log data
Using Spark to process data
Integrating Kafka to complex event alert
and more...
Web Server Log Processing using Hadoop
In this hadoop project, you will be using a sample application log file from an application server to a demonstrated scaled-down server log processing pipeline.
Explore hive usage efficiently in this hadoop hive project using various file formats such as JSON, CSV, ORC, AVRO and compare their relative performances
Common misuse/abuse of hive
How to use and interpret Hive's explain command
File formats and their relative performance (Text, JSON, SequenceFile, Avro, ORC and Parquet)
Compression
Spark and hive for transformation
and more...
Tough engineering choices with large datasets in Hive Part - 1
Explore hive usage efficiently in this hadoop hive project using various file formats such as JSON, CSV, ORC, AVRO and compare their relative performances
In this hadoop project, you will be using a sample application log file from an application server to a demonstrated scaled-down server log processing pipeline.
The benefits of log-mining in certain industries
A full log-mining application use-case
Using Flume to ingest log data
Using Spark to process data
Integrating Kafka to complex event alert
and more...
Web Server Log Processing using Hadoop
In this hadoop project, you will be using a sample application log file from an application server to a demonstrated scaled-down server log processing pipeline.
In this hadoop project, we are going to be continuing the series on data engineering by discussing and implementing various ways to solve the hadoop small file problem.
What is Small file problem in Hadoop
How it arises (Batch and Streaming mode)
Solution (Streaming): Using flume
Solution (Streaming): Preprocessing and storing in a NoSQL database
Solution (Batch): Merging before storing in HDFS
and more...
Online Hadoop Projects -Solving small file problem in Hadoop
In this hadoop project, we are going to be continuing the series on data engineering by discussing and implementing various ways to solve the hadoop small file problem.
Spark Project - Discuss real-time monitoring of taxis in a city. The real-time data streaming will be simulated using Flume. The ingestion will be done using Spark Streaming.
Key-Value (NoSQL) Databases
Using Redis as a pub/sub message-oriented middleware
Using Redis as a caching server/persistence store
Streaming data with Flume/Spark integration
Real-time processing and display of streamed data on a "dashboard".
and more...
Real-time Auto Tracking with Spark-Redis
Spark Project - Discuss real-time monitoring of taxis in a city. The real-time data streaming will be simulated using Flume. The ingestion will be done using Spark Streaming.
In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security
Making a case for real time processing of log files
Getting logs at real time using Flume Log4J appenders
Making a case for Kafka for log aggregation.
Storing log event as a time series datasets in HBase
Integrating Hive and HBase for data retrieval using query.
and more...
Real-Time Log Processing using Spark Streaming Architecture
In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security
The goal of this Spark project is to analyze business reviews from Yelp dataset and ingest the final output of data processing in Elastic Search.Also, use the visualisation tool in the ELK stack to visualize various kinds of ad-hoc reports from the data.
Ingesting data from relational database using Sqoop
Ingesting data from relational database directly into Spark
Processing relational data in Spark
Ingesting processed data into Elasticsearch
Visualizing review analytics using Kibana
Spark Project-Analysis and Visualization on Yelp Dataset
The goal of this Spark project is to analyze business reviews from Yelp dataset and ingest the final output of data processing in Elastic Search.Also, use the visualisation tool in the ELK stack to visualize various kinds of ad-hoc reports from the data.
In this big data project, we will continue from a previous hive project "Data engineering on Yelp Datasets using Hadoop tools" and do the entire data processing using spark.
Doing data processing using Spark
Normalizing and denormalizing dataset into hive tables
Various ways of integrating Hive and Spark
Various complex data structures in Hive through spark
Exporting some of the processed datasets to RDBMS
Yelp Data Processing Using Spark And Hive Part 1
In this big data project, we will continue from a previous hive project "Data engineering on Yelp Datasets using Hadoop tools" and do the entire data processing using spark.
In this hive project , we will build a Hive data warehouse from a raw dataset stored in HDFS and present the data in a relational structure so that querying the data will be natural.
• How to run hive queries on Spark
• Hadoop data warehousing with Hive
• Using the interactive Scala Build Tool (sbt) with Spark
• Data serialization with kryo serialization example
• Performance optimization using caching.
and more...
Building a Data Warehouse using Spark on Hive
In this hive project , we will build a Hive data warehouse from a raw dataset stored in HDFS and present the data in a relational structure so that querying the data will be natural.
In this big data project, we will talk about Apache Zeppelin. We will write code, write notes, build charts and share all in one single data analytics environment using Hive, Spark and Pig.
Apache Zeppelin: What it is and how it works
Installing Zeppelin interpreters
Running Spark, Hive and Pig code on your notebook
Writing markdown notes or narrative text.
Collaboration or Sharing your book with others
and more...
Data Analysis and Visualisation using Spark and Zeppelin
In this big data project, we will talk about Apache Zeppelin. We will write code, write notes, build charts and share all in one single data analytics environment using Hive, Spark and Pig.
In this big data project, we will look at how to mine and make sense of connections in a simple way by building a Spark GraphX Algorithm and a Network Crawler.
Designing your Github network of persons
Building the network model in HBase
Running your network crawler
Using spark to analyze the network
Running graph algorithms using GraphFrame or Spark GraphX
Design a Network Crawler by Mining Github Social Profiles
In this big data project, we will look at how to mine and make sense of connections in a simple way by building a Spark GraphX Algorithm and a Network Crawler.
In this Neo4j project, you will do network analysis using a graph database to find patterns on how a social network affects business reviews and ratings.
Introduce key terminologies in graph database
Short introduction to cypher
Spark-Neo4j connector
Introduction to Spark GraphX
Data analysis using GraphX and Neo4j
Neo4j Project using Yelp dataset to analyse ratings from users
In this Neo4j project, you will do network analysis using a graph database to find patterns on how a social network affects business reviews and ratings.
The goal of this spark project is to analyse the level and strength of interactions across areas of coverage of a telecom provider between different areas in the city of Milan.
Introduction to Graphs
Introduction to Spark GraphX
Building a graph structure from our dataset
Use of Spark GraphX graph operators
Graph Visualization
and more...
Analysis of Community Interactions using Spark GraphX
The goal of this spark project is to analyse the level and strength of interactions across areas of coverage of a telecom provider between different areas in the city of Milan.
The goal of this apache kafka project is to process log entries from applications in real-time using Kafka for the streaming architecture in a microservice sense.
Re-state the case for real-time processing of log files
Run through our application and real-time log collection using Flume Log4J appenders
Flume Kafka Integration (Channels or Sink)
Kafka Stream
Kafka Connect
and more...
Real-Time Log Processing in Kafka for Streaming Architecture
The goal of this apache kafka project is to process log entries from applications in real-time using Kafka for the streaming architecture in a microservice sense.
Spark Project - Discuss real-time monitoring of taxis in a city. The real-time data streaming will be simulated using Flume. The ingestion will be done using Spark Streaming.
Key-Value (NoSQL) Databases
Using Redis as a pub/sub message-oriented middleware
Using Redis as a caching server/persistence store
Streaming data with Flume/Spark integration
Real-time processing and display of streamed data on a "dashboard".
and more...
Real-time Auto Tracking with Spark-Redis
Spark Project - Discuss real-time monitoring of taxis in a city. The real-time data streaming will be simulated using Flume. The ingestion will be done using Spark Streaming.
In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security
Making a case for real time processing of log files
Getting logs at real time using Flume Log4J appenders
Making a case for Kafka for log aggregation.
Storing log event as a time series datasets in HBase
Integrating Hive and HBase for data retrieval using query.
and more...
Real-Time Log Processing using Spark Streaming Architecture
In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security
The goal of this Spark project is to analyze business reviews from Yelp dataset and ingest the final output of data processing in Elastic Search.Also, use the visualisation tool in the ELK stack to visualize various kinds of ad-hoc reports from the data.
Ingesting data from relational database using Sqoop
Ingesting data from relational database directly into Spark
Processing relational data in Spark
Ingesting processed data into Elasticsearch
Visualizing review analytics using Kibana
Spark Project-Analysis and Visualization on Yelp Dataset
The goal of this Spark project is to analyze business reviews from Yelp dataset and ingest the final output of data processing in Elastic Search.Also, use the visualisation tool in the ELK stack to visualize various kinds of ad-hoc reports from the data.
In this big data project, we will talk about Apache Zeppelin. We will write code, write notes, build charts and share all in one single data analytics environment using Hive, Spark and Pig.
Apache Zeppelin: What it is and how it works
Installing Zeppelin interpreters
Running Spark, Hive and Pig code on your notebook
Writing markdown notes or narrative text.
Collaboration or Sharing your book with others
and more...
Data Analysis and Visualisation using Spark and Zeppelin
In this big data project, we will talk about Apache Zeppelin. We will write code, write notes, build charts and share all in one single data analytics environment using Hive, Spark and Pig.
In this big data project, we'll work with Apache Airflow and write scheduled workflow, which will download data from Wikipedia archives, upload to S3, process them in HIVE and finally analyze on Zeppelin Notebooks.
Workflows and their uses
Apache Airflow
Working with Qubole and S3
Hive table creation and data processing
Charting via Zeppelin Notebooks
Big Data Hadoop Project-Visualize Daily Wikipedia Trends
In this big data project, we'll work with Apache Airflow and write scheduled workflow, which will download data from Wikipedia archives, upload to S3, process them in HIVE and finally analyze on Zeppelin Notebooks.
The goal of this apache kafka project is to process log entries from applications in real-time using Kafka for the streaming architecture in a microservice sense.
Re-state the case for real-time processing of log files
Run through our application and real-time log collection using Flume Log4J appenders
Flume Kafka Integration (Channels or Sink)
Kafka Stream
Kafka Connect
and more...
Real-Time Log Processing in Kafka for Streaming Architecture
The goal of this apache kafka project is to process log entries from applications in real-time using Kafka for the streaming architecture in a microservice sense.
In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security
Making a case for real time processing of log files
Getting logs at real time using Flume Log4J appenders
Making a case for Kafka for log aggregation.
Storing log event as a time series datasets in HBase
Integrating Hive and HBase for data retrieval using query.
and more...
Real-Time Log Processing using Spark Streaming Architecture
In this Spark project, we are going to bring processing to the speed layer of the lambda architecture which opens up capabilities to monitor application real time performance, measure real time comfort with applications and real time alert in case of security
In this hadoop project, you will be using a sample application log file from an application server to a demonstrated scaled-down server log processing pipeline.
The benefits of log-mining in certain industries
A full log-mining application use-case
Using Flume to ingest log data
Using Spark to process data
Integrating Kafka to complex event alert
and more...
Web Server Log Processing using Hadoop
In this hadoop project, you will be using a sample application log file from an application server to a demonstrated scaled-down server log processing pipeline.
The goal of this IoT project is to build an argument for generalized streaming architecture for reactive data ingestion based on a microservice architecture.
Streaming architectures (Lambda & Kappa)
MQTT and the IOT
Making decision about MQTT-Spark Streaming vs Kafka-Spark Streaming
Using Kafka as a data hub for streaming architecture
HBase and Spark Integration using the Spark HBase connector
and more...
IoT Project-Learn to design an IoT Ready Infrastructure
The goal of this IoT project is to build an argument for generalized streaming architecture for reactive data ingestion based on a microservice architecture.
In this Neo4j project, you will do network analysis using a graph database to find patterns on how a social network affects business reviews and ratings.
Introduce key terminologies in graph database
Short introduction to cypher
Spark-Neo4j connector
Introduction to Spark GraphX
Data analysis using GraphX and Neo4j
Neo4j Project using Yelp dataset to analyse ratings from users
In this Neo4j project, you will do network analysis using a graph database to find patterns on how a social network affects business reviews and ratings.
In this big data project using Neo4j, we will be remodelling the movielens dataset in a graph structure and using that structures to answer questions in different ways.
Thinking in terms of Graphs
Data modeling in a Graph database
Neo4J and the CAP theorem
Discuss possible scenarios of integrating Neo4J with a big data pipeline.
Migrating data from CSV to Neo4J database
and more...
Modeling & Thinking in Graphs(Neo4J) using Movielens Dataset
In this big data project using Neo4j, we will be remodelling the movielens dataset in a graph structure and using that structures to answer questions in different ways.
Spark Project - Discuss real-time monitoring of taxis in a city. The real-time data streaming will be simulated using Flume. The ingestion will be done using Spark Streaming.
Key-Value (NoSQL) Databases
Using Redis as a pub/sub message-oriented middleware
Using Redis as a caching server/persistence store
Streaming data with Flume/Spark integration
Real-time processing and display of streamed data on a "dashboard".
and more...
Real-time Auto Tracking with Spark-Redis
Spark Project - Discuss real-time monitoring of taxis in a city. The real-time data streaming will be simulated using Flume. The ingestion will be done using Spark Streaming.
Every year, people looking to begin their big data career run into a familiar conundrum -
"How can I land a big data job with limited experience in this field?".
For an emerging field like big data, finding internships or full-time big data jobs requires you to showcase relevant achievements working with popular open source big data tools like, Hadoop, Spark, Kafka, Pig, Hive, and more. Big data and project-based learning are a perfect fit. The best way to get started is to begin working on diverse big data project titles under the mentorship of industry experts. Professionals will love working on these big data projects because it's like a secret. There is so much practical learning involved you don't realize it. DeZyre's big data projects are perfect for beginners, college students, engineering students, professionals wanting to make a career switch and anyone who wants to master big data skills with hands-on experience.
Big Data Projects for Beginners
If you have graduate degree in analytics or relevant field from a top-tier college, it is easy for you to get a big data job. Employers believe that you will be able to add value to their business because of the prestige of the college that has awarded you the degree, and the reality that it is in a subject that is relevant to the kind of skills they are looking for. If you do not have an analytics degree from a top-tier college then you need to build that trust yourself that you have the big data skills that the employer is looking for. The best way to build trust with the hiring manager is to work on interesting big data project ideas and build a portfolio of multiple big data projects - Hadoop projects, spark projects, hive projects, Kafka projects, impala projects, and more. The more "real-world" the big data projects are, the more the hiring manager will trust that you will be an asset to their organization , and the greater are your chances of landing the big data job. The best thing about big data careers is that the work you do on building diverse big data projects often looks exactly similar to the work you will do once you are hired.
For IT professionals or anybody with basic big data knowledge, Dezyre's mini projects on big data will help them take responsibility in solving challenging data problems, and help gain expertise on the popular big data tools like Hadoop, Spark, Hive, Pig,
The good news for people in search of big data projects for CSE students is that there are couple of websites that have big data projects with source code. If you google for search terms like "big data projects GitHub" or "big data projects Quora", you might find suggestions on multiple big data project titles, however, for students on the hunt for big data final year projects, titles and source code is not what all they need for learning. Students need industry expert guidance for deeper understanding and greater retention of knowledge so that they can apply what they know to new real-world big data problems. DeZyre has an excellent project-based learning platform where students will enjoy using a spectrum of big data tools under expert guidance.
Here are some popular big data project titles among the college students-
IT professionals and college students rate our big data projects as exceptional. Whether you are looking to upgrade your skills or you are looking to learn about the complete end-to-end implementation of various big data tools like Hadoop, spark, pig , hive, Kafka, and more, Dezyre's mini projects on big data are just what you want.
What will you get when you enroll for DeZyres Big Data projects?
Big Data Project Source Code: Examine and implement end-to-end real-world big data projects from the Banking, eCommerce, and Entertainment sector using this source code.
Recorded Demo: Watch a video explanation on how to execute these big data projects.
Complete Solution Kit: Get access to the solution design, documents, and supporting reference material, if any.
Mentor Support: Get your technical questions answered with mentorship from the best industry experts for a nominal fee.
Hands-On Knowledge: Equip yourself with practical skills on the big data ecosystem.
