Movielens dataset analysis for movie recommendations using Spark in Azure

Movielens dataset analysis for movie recommendations using Spark in Azure

In this Databricks Azure tutorial project, you will use Spark Sql to analyse the movielens dataset to provide movie recommendations. As part of this you will deploy Azure data factory, data pipelines and visualise the analysis.
explanation image

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

ipython image

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

project experience

Project Experience

Add project experience to your Linkedin/Github profiles.

Customer Love

Read All Reviews
profile image

Mike Vogt linkedin profile url

Information Architect at Bank of America

I have had a very positive experience. The platform is very rich in resources, and the expert was thoroughly knowledgeable on the subject matter - real world hands-on experience. I wish I had this... Read More

profile image

Ray Han linkedin profile url

Tech Leader | Stanford / Yale University

I think that they are fantastic. I attended Yale and Stanford and have worked at Honeywell,Oracle, and Arthur Andersen(Accenture) in the US. I have taken Big Data and Hadoop,NoSQL, Spark, Hadoop... Read More

What will you learn

Understanding the problem statement & Microsoft Azure Platform
Developing end to end data pipeline using Microsoft Azure and Databricks Spark
Getting Microsoft Azure Subscription
Creating Resource Group
Introduction to Storage Account
Upload raw data to cloud
Introduction to Azure Data Factory
Create and run ADF pipelines
Introduction to Azure Databricks
Spinning up Databricks cluster
Read data from storage account
Writing Spark Sql on Databricks
Data analysis using spark on Databricks
Data cleansing using spark on Databricks
Data Transformation using spark
Movie Recommendation algorithm using Spark in Azure
Model deployment creating FlaskAPI

Project Description

A movie recommendation system is used by top streaming services like Netflix, Amazon Prime, Hulu, Hotstar etc to recommend movies to their users based on historical viewing patterns.

Before the final recommendation is made, there is a complex data pipeline that brings data from many sources to the recommendation engine. In this project, we use Databricks Spark on Azure with Spark Sql to build this data pipeline.

Our dataset is from GroupLens Research, which is a research group in the Department of Computer Science and Engineering at the University of Minnesota. They operate a movie recommender based on collaborative filtering called MovieLens. This dataset (ml-latest) describes 5-star rating and free-text tagging activity from MovieLens, a movie recommendation service. It contains 22884377 ratings and 586994 tag applications across 34208 movies. These data were created by 247753 users between January 09, 1995 and January 29, 2016. This dataset was generated on January 29, 2016.

Similar Projects

Data Analysis and Visualisation using Spark and Zeppelin

In this big data project, we will talk about Apache Zeppelin. We will write code, write notes, build charts and share all in one single data analytics environment using Hive, Spark and Pig.

View Project Details

Tough engineering choices with large datasets in Hive Part - 2

This is in continuation of the previous Hive project "Tough engineering choices with large datasets in Hive Part - 1", where we will work on processing big data sets using Hive.

View Project Details

Analyse Yelp Dataset with Spark & Parquet Format on Azure Databricks

In this Databricks Azure project, you will use Spark & Parquet file formats to analyse the Yelp reviews dataset. As part of this you will deploy Azure data factory, data pipelines and visualise the analysis.

View Project Details

Curriculum For This Mini Project

Introduction
04m
Signing Up To Microsoft Azure Cloud
02m
Create A Resource Group In Azure
02m
Setting Up Azure Storage Account
04m
Uploading Raw Data
05m
Setup Azure Data Factory
12m
Run The Adf Pipeline
07m
Introduction To Azure Databricks
03m
Setting Up A Cluster In Azure Databricks
04m
Authorise Storage Account In Databricks
03m
Reading Data From Databricks
02m
Exploring The Dataset Using Pyspark
08m
Data Transformation And Analysis Using Pyspark
08m
Pyspark Data Analysis - 1
11m
Pyspark Data Analysis - 2
11m