Movielens dataset analysis for movie recommendations using Spark in Azure

In this Databricks Azure tutorial project, you will use Spark Sql to analyse the movielens dataset to provide movie recommendations. As part of this you will deploy Azure data factory, data pipelines and visualise the analysis.

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

What will you learn

Understanding the problem statement & Microsoft Azure Platform

Developing end to end data pipeline using Microsoft Azure and Databricks Spark

Getting Microsoft Azure Subscription

Creating Resource Group

Introduction to Storage Account

Upload raw data to cloud

Introduction to Azure Data Factory

Create and run ADF pipelines

Introduction to Azure Databricks

Spinning up Databricks cluster

Read data from storage account

Writing Spark Sql on Databricks

Data analysis using spark on Databricks

Data cleansing using spark on Databricks

Data Transformation using spark

Movie Recommendation algorithm using Spark in Azure

Model deployment creating FlaskAPI

Project Description

A movie recommendation system is used by top streaming services like Netflix, Amazon Prime, Hulu, Hotstar etc to recommend movies to their users based on historical viewing patterns.

Before the final recommendation is made, there is a complex data pipeline that brings data from many sources to the recommendation engine. In this project, we use Databricks Spark on Azure with Spark Sql to build this data pipeline.

Our dataset is from GroupLens Research, which is a research group in the Department of Computer Science and Engineering at the University of Minnesota. They operate a movie recommender based on collaborative filtering called MovieLens. This dataset (ml-latest) describes 5-star rating and free-text tagging activity from MovieLens, a movie recommendation service. It contains 22884377 ratings and 586994 tag applications across 34208 movies. These data were created by 247753 users between January 09, 1995 and January 29, 2016. This dataset was generated on January 29, 2016.

Similar Projects

Design a Network Crawler by Mining Github Social Profiles

In this big data project, we will look at how to mine and make sense of connections in a simple way by building a Spark GraphX Algorithm and a Network Crawler.

View Project Details

Tough engineering choices with large datasets in Hive Part - 2

This is in continuation of the previous Hive project "Tough engineering choices with large datasets in Hive Part - 1", where we will work on processing big data sets using Hive.

View Project Details

Implementing OLAP on Hadoop using Apache Kylin

In this big data project, we will be performing an OLAP cube design using AdventureWorks database. The deliverable for this session will be to design a cube, build and implement it using Kylin, query the cube and even connect familiar tools (like Excel) with our new cube.

View Project Details

Curriculum For This Mini Project

Introduction

04m

Signing Up To Microsoft Azure Cloud

02m

Create A Resource Group In Azure

02m

Setting Up Azure Storage Account

04m

Uploading Raw Data

05m

Setup Azure Data Factory

12m

Run The Adf Pipeline

07m

Introduction To Azure Databricks

03m

Setting Up A Cluster In Azure Databricks

04m

Authorise Storage Account In Databricks

03m

Reading Data From Databricks

02m

Exploring The Dataset Using Pyspark

08m

Data Transformation And Analysis Using Pyspark

08m

Pyspark Data Analysis - 1

11m

Pyspark Data Analysis - 2

11m