Predict Quora Question Pairs Meaning using NLP in Python

Predict Quora Question Pairs Meaning using NLP in Python

The goal of this NLP project is to predict which of the provided quora question pairs contain two questions with the same meaning.

Videos

Each project comes with 2-5 hours of micro-videos explaining the solution.

Code & Dataset

Get access to 50+ solved projects with iPython notebooks and datasets.

Project Experience

Add project experience to your Linkedin/Github profiles.

Customer Love

Read All Reviews

Mohamed Yusef Ahmed

Software Developer at Taske

Recently I became interested in Hadoop as I think its a great platform for storing and analyzing large structured and unstructured data sets. The experts did a great job not only explaining the... Read More

Arvind Sodhi

VP - Data Architect, CDO at Deutsche Bank

I have extensive experience in data management and data processing. Over the past few years I saw the data management technology transition into the Big Data ecosystem and I needed to follow suit. I... Read More

What will you learn

Understanding the problem statement
Initializing necessary libraries and understanding its use
Difference between NLU and NLG
Constructing features that we will later use to embed our samples with
Visualizing distribution of Target variable using Barplot
Plotting Log Histogram
Visualizing distribution of corpus using Barplot
Normalized histogram of character count in questions
t-Distributed Stochastic Neighbor Embedding for dimensionality reduction
Stemming, Lemmatization and removing stopwords
What is Tokenization,N-grams and POS tagging
Converting Word to Vectors and making it ready for training
Calculating Cosine Similarity between different words
Applying Logistic Regression
Applying ensemble model Random Forest Classifier
Applying Gradient Boosting Classifier
Performing Train_test_split
Applying Grid search CV for hyperparameter tuning
Defining F1 metrics and Confusion Metrics
Modularizing the code using Pipeline

Project Description

Overview:

Where else but Quora can a physicist helps a chef with a math problem and gets cooking tips in return? Quora is a place to gain and share knowledge—about anything. It’s a platform to ask questions and connect with people who contribute unique insights and quality answers. This empowers people to learn from each other and to better understand the world.

Over 100 million people visit Quora every month, so it's no surprise that many people ask similarly worded questions. Multiple questions with the same intent can cause seekers to spend more time finding the best answer to their question and make writers feel they need to answer multiple versions of the same question. Quora values canonical questions because they provide a better experience to active seekers and writers, and offer more value to both of these groups in the long term.

Currently, Quora uses a Random Forest model to identify duplicate questions. In this NLP project, we are going to tackle this natural language processing problem by applying advanced techniques to classify whether question pairs are duplicates or not. Doing so will make it easier to find high-quality answers to questions resulting in an improved experience for Quora writers, seekers, and readers.

Data Introduction:

The goal of this NLP project in Python is to predict which of the provided pairs of questions contain two questions with the same meaning. The ground truth is the set of labels that have been supplied by human experts. The ground truth labels are inherently subjective, as the true meaning of sentences can never be known with certainty. Human labeling is also a 'noisy' process, and reasonable people will disagree. As a result, the ground truth labels on this dataset should be taken to be 'informed' but not 100% accurate, and may include incorrect labeling. We believe the labels, on the whole, to represent a reasonable consensus, but this may often not be true on a case by case basis for individual items in the dataset.

Similar Projects

In this machine learning churn project, we implement a churn prediction model in python using ensemble techniques.

Data Science Project-Predict the car insurance policy a customer buys after receiving a number of quotes.

In this data science project, you will contextualize customer data and predict the likelihood a customer will stay at 100 different hotel groups.

Curriculum For This Mini Project

7-Apr-2017
02h 41m
8-Apr-2017
02h 06m