Predict Quora Question Pairs Meaning using NLP in Python

Predict Quora Question Pairs Meaning using NLP in Python

The goal of this NLP project is to predict which of the provided quora question pairs contain two questions with the same meaning.

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

Customer Love

Read All Reviews

Nathan Elbert

Senior Data Scientist at Tiger Analytics

This was great. The use of Jupyter was great. Prior to learning Python I was a self taught SQL user with advanced skills. I hold a Bachelors in Finance and have 5 years of business experience.. I... Read More

SUBHABRATA BISWAS

Lead Consultant, ITC Infotech

The project orientation is very much unique and it helps to understand the real time scenarios most of the industries are dealing with. And there is no limit, one can go through as many projects... Read More

What will you learn

Understanding the problem statement
Initializing necessary libraries and understanding its use
Difference between NLU and NLG
Constructing features that we will later use to embed our samples with
Visualizing distribution of Target variable using Barplot
Plotting Log Histogram
Visualizing distribution of corpus using Barplot
Normalized histogram of character count in questions
t-Distributed Stochastic Neighbor Embedding for dimensionality reduction
Stemming, Lemmatization and removing stopwords
What is Tokenization,N-grams and POS tagging
Converting Word to Vectors and making it ready for training
Calculating Cosine Similarity between different words
Applying Logistic Regression
Applying ensemble model Random Forest Classifier
Applying Gradient Boosting Classifier
Performing Train_test_split
Applying Grid search CV for hyperparameter tuning
Defining F1 metrics and Confusion Metrics
Modularizing the code using Pipeline

Project Description

Overview:

Where else but Quora can a physicist helps a chef with a math problem and gets cooking tips in return? Quora is a place to gain and share knowledge—about anything. It’s a platform to ask questions and connect with people who contribute unique insights and quality answers. This empowers people to learn from each other and to better understand the world.

Over 100 million people visit Quora every month, so it's no surprise that many people ask similarly worded questions. Multiple questions with the same intent can cause seekers to spend more time finding the best answer to their question and make writers feel they need to answer multiple versions of the same question. Quora values canonical questions because they provide a better experience to active seekers and writers, and offer more value to both of these groups in the long term.

Currently, Quora uses a Random Forest model to identify duplicate questions. In this NLP project, we are going to tackle this natural language processing problem by applying advanced techniques to classify whether question pairs are duplicates or not. Doing so will make it easier to find high-quality answers to questions resulting in an improved experience for Quora writers, seekers, and readers.

Data Introduction:

The goal of this NLP project in Python is to predict which of the provided pairs of questions contain two questions with the same meaning. The ground truth is the set of labels that have been supplied by human experts. The ground truth labels are inherently subjective, as the true meaning of sentences can never be known with certainty. Human labeling is also a 'noisy' process, and reasonable people will disagree. As a result, the ground truth labels on this dataset should be taken to be 'informed' but not 100% accurate, and may include incorrect labeling. We believe the labels, on the whole, to represent a reasonable consensus, but this may often not be true on a case by case basis for individual items in the dataset.

Similar Projects

In this project, we are going to talk about insurance forecast by using regression techniques.

The goal of this machine learning project is to predict which products existing customers will use next month based on their past behaviour and that of similar customers.

Given a customer's search query and the returned product in text format, your predictive model needs to tell whether it is what the customer was looking for.

Curriculum For This Mini Project

7-Apr-2017
02h 41m
8-Apr-2017
02h 06m