Predict Quora Question Pairs Meaning using NLP in Python

Predict Quora Question Pairs Meaning using NLP in Python

The goal of this NLP project is to predict which of the provided quora question pairs contain two questions with the same meaning.

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

What will you learn

Understanding the problem statement
Initializing necessary libraries and understanding its use
Difference between NLU and NLG
Constructing features that we will later use to embed our samples with
Visualizing distribution of Target variable using Barplot
Plotting Log Histogram
Visualizing distribution of corpus using Barplot
Normalized histogram of character count in questions
t-Distributed Stochastic Neighbor Embedding for dimensionality reduction
Stemming, Lemmatization and removing stopwords
What is Tokenization,N-grams and POS tagging
Converting Word to Vectors and making it ready for training
Calculating Cosine Similarity between different words
Applying Logistic Regression
Applying ensemble model Random Forest Classifier
Applying Gradient Boosting Classifier
Performing Train_test_split
Applying Grid search CV for hyperparameter tuning
Defining F1 metrics and Confusion Metrics
Modularizing the code using Pipeline

Project Description

Overview:

Where else but Quora can a physicist helps a chef with a math problem and gets cooking tips in return? Quora is a place to gain and share knowledge—about anything. It’s a platform to ask questions and connect with people who contribute unique insights and quality answers. This empowers people to learn from each other and to better understand the world.

Over 100 million people visit Quora every month, so it's no surprise that many people ask similarly worded questions. Multiple questions with the same intent can cause seekers to spend more time finding the best answer to their question and make writers feel they need to answer multiple versions of the same question. Quora values canonical questions because they provide a better experience to active seekers and writers, and offer more value to both of these groups in the long term.

Currently, Quora uses a Random Forest model to identify duplicate questions. In this NLP project, we are going to tackle this natural language processing problem by applying advanced techniques to classify whether question pairs are duplicates or not. Doing so will make it easier to find high-quality answers to questions resulting in an improved experience for Quora writers, seekers, and readers.

Data Introduction:

The goal of this NLP project in Python is to predict which of the provided pairs of questions contain two questions with the same meaning. The ground truth is the set of labels that have been supplied by human experts. The ground truth labels are inherently subjective, as the true meaning of sentences can never be known with certainty. Human labeling is also a 'noisy' process, and reasonable people will disagree. As a result, the ground truth labels on this dataset should be taken to be 'informed' but not 100% accurate, and may include incorrect labeling. We believe the labels, on the whole, to represent a reasonable consensus, but this may often not be true on a case by case basis for individual items in the dataset.

Similar Projects

This data science in python project predicts if a loan should be given to an applicant or not. We predict if the customer is eligible for loan based on several factors like credit score and past history.

Build a predictive model to correctly classify products between 9 product categories (fashion, electronics, etc.) using the Otto Group dataset.

In this data science project, we will look at few examples where we can apply various time series forecasting techniques.

Curriculum For This Mini Project

7-Apr-2017
02h 41m
8-Apr-2017
02h 06m