Hadoop Project-Analysis of Yelp Dataset using Hadoop Hive

Hadoop Project-Analysis of Yelp Dataset using Hadoop Hive

The goal of this hadoop project is to apply some data engineering principles to Yelp Dataset in the areas of processing, storage, and retrieval.
explanation image

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

ipython image

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

project experience

Project Experience

Add project experience to your Linkedin/Github profiles.

Customer Love

Read All Reviews
profile image

Dhiraj Tandon linkedin profile url

Solution Architect-Cyber Security at ColorTokens

My Interaction was very short but left a positive impression. I enrolled and asked for a refund since I could not find the time. What happened next: They initiated Refund immediately. Their... Read More

profile image

Swati Patra linkedin profile url

Systems Advisor , IBM

I have 11 years of experience and work with IBM. My domain is Travel, Hospitality and Banking - both sectors process lots of data. The way the projects were set up and the mentors' explanation was... Read More

What will you learn

Understanding Data Engineering, different roles, and tools used
Understanding the Yelp dataset
What is Dataset schema and how to create your own schema
Tools to be used during data processing
What is Hadoop small file problem and how to solve them
Understanding Hadoop Small file problem using and example
How to use the Online system instead of Batch System
Serving layer vs Batch layer (Neo4j vs HDFS)
Data Sampling and Understanding
Understanding database tables and creating them in HDFS
Provisioning access to data using hive/impala
Selecting Parquet or Avro for creating Schemes for my Data
Performing Data analysis and Data modeling on the dataset
Solving Complex Cases in Hadoop

Project Description

Data engineering is the science of acquiring, aggregating or collection, processing, and storage of data either in batch or in real-time as well as providing the variety of means of serving these data to other users which could include a data scientist. It involves software engineering practices on big data.

The goal of this big data project is apply data engineering principles to the Yelp Dataset in the areas of processing, storage, and retrieval. We will not include data ingestion since we are already downloading the data from the yelp challenge website.

Similar Projects

In this spark streaming project, we are going to build the backend of a IT job ad website by streaming data from twitter for analysis in spark.

Learn to perform 1) Twitter Sentiment Analysis using Spark Streaming, NiFi and Kafka, and 2) Build an Interactive Data Visualization for the analysis using Python Plotly.

In this PySpark project, you will simulate a complex real-world data pipeline based on messaging. This project is deployed using the following tech stack - NiFi, PySpark, Hive, HDFS, Kafka, Airflow, Tableau and AWS QuickSight.

Curriculum For This Mini Project

Overview
05m
What-is-Data-Engineering ?
08m
The Yelp Dataset
03m
Dataset schema and Job roles
13m
Data format and storage
07m
Data processing tools
10m
Hadoop small file problem
15m
Example - Hadoop small file problem
08m
Data provisioning
07m
Data sampling and understanding - 1
14m
Create database tables
05m
Parquet versus Avro
19m
Data Analysis
36m
Data Modelling
29m
Complex cases
16m