Process a Million Song Dataset to Predict Song Preferences

In this big data project, we will discover songs for those artists that are associated with the different cultures across the globe.
Videos
Each project comes with 2-5 hours of micro-videos explaining the solution.
Code & Dataset
Get access to 50+ solved projects with iPython notebooks and datasets.
Project Experience
Add project experience to your Linkedin/Github profiles.

What will you learn

  • Analysis large datasets easily and efficiently
  • Using data flow programming language "Pig Latin" for analysis
  • Data compression using LZO codec
  • PigLatin UDF "DataFu" (Created by LinkedIn) for data localization
  • Working with Hierarchical Data Format (HDF5)

Project Description

This big data hadoop project aims at being the best possible offline evaluation of a music recommendation system.  Any type of algorithm can be used: collaborative filtering, content-based methods, web crawling. By relying on the Million Song Dataset, the data for this big data project is completely open: almost everything is known and possibly available.

What is the task in a few words? You have: 

  1. the full listening history for 1M users, 
  2. half of the listening history for 110K users (10K validation set, 100K test set), 

and you must predict the missing half. How much easier can it get?

The most straightforward approach to this task is pure collaborative filtering, but remember that there is a wealth of information available to you through the Million Song Dataset.  For Million Song Dataset Download, click this link - labrosa.ee.columbia.edu/millionsong/. Go ahead, explore!

Curriculum For This Mini Project

 
  13-Jul-2016
02h 35m
  14-Jul-2016
02h 41m