Process a Million Song Dataset to Predict Song Preferences

Process a Million Song Dataset to Predict Song Preferences

In this big data project, we will discover songs for those artists that are associated with the different cultures across the globe.


Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

Customer Love

Read All Reviews

Prasanna Lakshmi T

Advisory System Analyst at IBM

Initially, I was unaware of how this would cater to my career needs. But when I stumbled through the reviews given on the website. I went through many of them and found them all positive. I would... Read More

James Peebles

Data Analytics Leader, IQVIA

This is one of the best of investments you can make with regards to career progression and growth in technological knowledge. I was pointed in this direction by a mentor in the IT world who I highly... Read More

What will you learn

Roadmap of the project
Horizontal Scalability of Hadoop and vertical scalability of RDMS
Pig Local and MapReduce working format , their explanation, and differences
Challenges in Pig MapReduce program
Overcoming Bandwidth challenges using Pig Tez
Analysis large datasets easily and efficiently
Understanding the Haversine formula and its application by Pig Latin UDF
Downloading the dataset and setting up Cloudera VMWare
PigLatin UDF "DataFu" (Created by LinkedIn) for data localization
Logging to the server using XShell-5 and Putty
Using data flow programming language "Pig Latin" for analysis
Using HDF5 for using as repository
Performing Basic EDA using Apache Ambari
Extracting Data from individual file and collectively pre-processing it
Registering UDF on Pig
Creating tables and using relational functions(Group, Join, CrossJoin, Filter, etc.)
Working with Hierarchical Data Format (HDF5)

Project Description

This big data hadoop project aims at being the best possible offline evaluation of a music recommendation system.  Any type of algorithm can be used: collaborative filtering, content-based methods, web crawling. By relying on the Million Song Dataset, the data for this big data project is completely open: almost everything is known and possibly available.

What is the task in a few words? You have: 

  1. the full listening history for 1M users, 
  2. half of the listening history for 110K users (10K validation set, 100K test set), 

and you must predict the missing half. How much easier can it get?

The most straightforward approach to this task is pure collaborative filtering, but remember that there is a wealth of information available to you through the Million Song Dataset.  For Million Song Dataset Download, click this link - Go ahead, explore!

Similar Projects

The goal of this hadoop project is to apply some data engineering principles to Yelp Dataset in the areas of processing, storage, and retrieval.

The goal of this IoT project is to build an argument for generalized streaming architecture for reactive data ingestion based on a microservice architecture. 

In this hive project, you will design a data warehouse for e-commerce environments.

Curriculum For This Mini Project

02h 35m
02h 41m