Data Mining Project on Yelp Dataset using Hadoop Hive

Use the Hadoop ecosystem to glean valuable insights from the Yelp dataset. You will be analyzing the different patterns that can be found in the Yelp data set, to come up with various approaches in solving a business problem.
Each project comes with 2-5 hours of micro-videos explaining the solution.
Code & Dataset
Get access to 50+ solved projects with iPython notebooks and datasets.
Project Experience
Add project experience to your Linkedin/Github profiles.

What will you learn

  • Analyze JSON dataLoading JSON format to Hive
  • Create a Schema to the fields in the table.
  • Creating queries to set up the EXTERNAL TABLE in Hive
  • Create new desired TABLE to copy the data.
  • Creating query to populate and filter the data.
  • Analyze log files in HIVE.

Project Description

Yelp it! is the term people use to review a local business, restaurant or products across the main US states and cities. Yelp has grown from a simple reviews site to something much more. It is now a strong community of users who contribute reviews per their own volition. Now let us understand what does this mean in terms of data that is generated in Yelp. Since its inception in 2004, Yelp has collected a staggering 25 million reviews for its local businesses, restaurants, doctors, services, etc. They have an average of 66 million unique visitors to their site every month. Yelp App is used on 5.7 million mobile devices. They have an impressive Y-O-Y growth with reviews growing by 64%, visitors growing at 67%, local businesses at 97% and active local advertisers at 118%. That is a LOT of data! Phew! 

It comes as no surprise when we say that Yelp has managed to crush all their competition mainly because they are so good at big data analysis. Data of this magnitude has a story to tell and businesses need to figure out what their data is telling them in order to make smarter business decisions than their competitors. In the following project, we have taken a Yelp data-set and we will be using Hive to analyze this data. Hive is the easiest of the Hadoop tools to learn. If you are from a data warehousing background and know SQL well - it will be a breeze to work on Hive. Hive is a data warehouse infrastructure built on top of Hadoop and is quite versatile in its usage, as it supports different storage types such as plain text, RCFile, Amazon S3, HBase, ORC, etc. Hive has its own SQL like language called HiveQL with schemas - which transparently converts queries to MapReduce or Apache Spark jobs. 

You will be working on solving these business problems for the end-user:

  • Overall business review counts for a particular period of time. 
  • Local Businesses' opening and closing timings. 
  • Amenities provided by each business. 
  • List the top restaurants in a state by the number of reviews.
  • List the top restaurants in number of listed categories.
  • Filter the top categories by number of review counts.
  • Joining reviews dataset with the businesses to do further analysis about the businesses. 


Curriculum For This Mini Project

  Dataset Overview
  Setting up the machine configuration
  Troubleshooting configuration
  Process flow of the solution
  Convert JSON data to text
  Convert JSON data to text - seperate fields from values
  Summary of Process flow
  Explanation of the Yelp dataset
  Load data into HDFS
  Create Database
  Working with the tables
  Working with map, strut and arrays
  Analyse Yelp reviews
  Analyse number of reviews by state and city
  Analyse top 10 categories by number of reviews
  Analyse top business which have over 2000 good reviews
  Analyse number of sub-categories available
  Analyse top 10 restaurants in each city with over 100 reviews
  Analyse top 50 restaurants by reviews that are open more than 11 hours/day
  Debugging query
  Ranking query
  Analyse top 3 businesses by average daily users, where users found the reviews useful
  Analyse increase in ratings for a business in a year