Predict Quora Question Pairs Meaning using NLP in Python

The goal of this NLP project is to predict which of the provided quora question pairs contain two questions with the same meaning.


Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

What will you learn

  • Understanding the problem statement

  • Initializing necessary libraries and understanding its use

  • Difference between NLU and NLG

  • Constructing features that we will later use to embed our samples with

  • Visualizing distribution of Target variable using Barplot

  • Plotting Log Histogram

  • Visualizing distribution of corpus using Barplot

  • Normalized histogram of character count in questions

  • t-Distributed Stochastic Neighbor Embedding for dimensionality reduction

  • Stemming, Lemmatization and removing stopwords

  • What is Tokenization,N-grams and POS tagging

  • Converting Word to Vectors and making it ready for training

  • Calculating Cosine Similarity between different words

  • Applying Logistic Regression

  • Applying ensemble model Random Forest Classifier

  • Applying Gradient Boosting Classifier

  • Performing Train_test_split

  • Applying Grid search CV for hyperparameter tuning

  • Defining F1 metrics and Confusion Metrics

  • Modularizing the code using Pipeline

Project Description


Where else but Quora can a physicist helps a chef with a math problem and gets cooking tips in return? Quora is a place to gain and share knowledge—about anything. It’s a platform to ask questions and connect with people who contribute unique insights and quality answers. This empowers people to learn from each other and to better understand the world.

Over 100 million people visit Quora every month, so it's no surprise that many people ask similarly worded questions. Multiple questions with the same intent can cause seekers to spend more time finding the best answer to their question and make writers feel they need to answer multiple versions of the same question. Quora values canonical questions because they provide a better experience to active seekers and writers, and offer more value to both of these groups in the long term.

Currently, Quora uses a Random Forest model to identify duplicate questions. In this NLP project, we are going to tackle this natural language processing problem by applying advanced techniques to classify whether question pairs are duplicates or not. Doing so will make it easier to find high-quality answers to questions resulting in an improved experience for Quora writers, seekers, and readers.

Data Introduction:

The goal of this NLP project in Python is to predict which of the provided pairs of questions contain two questions with the same meaning. The ground truth is the set of labels that have been supplied by human experts. The ground truth labels are inherently subjective, as the true meaning of sentences can never be known with certainty. Human labeling is also a 'noisy' process, and reasonable people will disagree. As a result, the ground truth labels on this dataset should be taken to be 'informed' but not 100% accurate, and may include incorrect labeling. We believe the labels, on the whole, to represent a reasonable consensus, but this may often not be true on a case by case basis for individual items in the dataset.

Similar Projects

Big Data Project Insurance Pricing Forecast Using Regression Analysis
In this project, we are going to talk about insurance forecast by using regression techniques.
Big Data Project Santander Product Recommendation ML Project in Python
The goal of this machine learning project is to predict which products existing customers will use next month based on their past behaviour and that of similar customers.
Big Data Project Music Recommendation System Project using Python and R
Machine Learning Project - Work with KKBOX's Music Recommendation System dataset to build the best music recommendation engine.
Big Data Project Bosch Production Line Performance Data Science Project
In this data science project, we will predict internal failures of Bosch using thousands of measurements and tests made for each component along the assembly line.

Curriculum For This Mini Project

02h 41m
02h 06m