How to find phrase and sentence similarity using Skip Gram model?
MACHINE LEARNING RECIPES DATA CLEANING PYTHON DATA MUNGING PANDAS CHEATSHEET     ALL TAGS

How to find phrase and sentence similarity using Skip Gram model?

How to find phrase and sentence similarity using Skip Gram model?

This recipe helps you find phrase and sentence similarity using Skip Gram model

Recipe Objective

How to find phrase and sentence similarity using Skip Gram model?

As we have discussed earlier Skip Gram, so it predicts the surrounding context words within a specific window given the current word. The input layer contains the current word and the output layer contains the setting words. The layer which is hidden contains the number of measurements wherein we need to do the representation of the current word present at the input layer.

Sentence and phrase similarity So these are terms in which we are going to detect the similarity between two sentences or phrases. In the event that equivalent words exist in both the sentences or expressions in the same request then we can say there is a sentence or phrase similarity, yet in the event that the words are in various order, at that point it should not completely consider a similar sentence or phrase. Let's understand this with practical implementation.

Step 1 - Import the necessary libraries

from nltk.tokenize import sent_tokenize, word_tokenize import warnings warnings.filterwarnings(action = 'ignore') import gensim from gensim.models import Word2Vec

Here we have imported the necessary packages along with the warnings and kept it as ignore because we know that there might be some warnings comming up when we run our program, but that can be ignored.

Step 2 - load the sample data

Sample = open("/content/alice_in_wonderland.txt","r") s = Sample.read()

Step 3 - Replace the escape character with spaces

f = s.replace("\n", " ")

Step 4 - Iterate and tokenize

import nltk nltk.download('punkt')
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
True
data = [] for i in sent_tokenize(f): temp = [] for j in word_tokenize(i): temp.append(j.lower()) data.append(temp)

Here we are taking a list as variable named data which is initially empty, after that we are going take a for loop which will iterate through each sentences present in the text file, and the second for loop will tokenize the sentences into words.

Step 5 - Create a Skip Gram model

model2 = gensim.models.Word2Vec(data, min_count = 1, size = 100, window = 5, sg = 1)

Step 6 - Print the result of Skip Gram model

print("Similarity between 'alice' " + "and 'girl' - Skip Gram : ", model2.similarity('alice', 'girl'),'\n') print("Similarity between 'alice' " + "and 'boy' - Skip Gram : ", model2.similarity('alice', 'boy'), '\n') print("Similarity between 'alice' " + "and 'rabbit' - Skip Gram : ",
Similarity between 'alice' and 'girl' - Skip Gram :  0.951415 

Similarity between 'alice' and 'boy' - Skip Gram :  0.94738436 

Similarity between 'alice' and 'rabbit' - Skip Gram :  0.97160363 

Similarity between 'girl' and 'rabbit' - Skip Gram :  0.98851126 

Similarity between 'girl' and 'cat' - Skip Gram :  0.9877583

Here from the above we can see the similarity between the words, we have used the words which are repeatedly used in the Sample text. So we can see the similarity between them by using skip gram model.

Relevant Projects

Ensemble Machine Learning Project - All State Insurance Claims Severity Prediction
In this ensemble machine learning project, we will predict what kind of claims an insurance company will get. This is implemented in python using ensemble machine learning algorithms.

Data Science Project-TalkingData AdTracking Fraud Detection
Machine Learning Project in R-Detect fraudulent click traffic for mobile app ads using R data science programming language.

Locality Sensitive Hashing Python Code for Look-Alike Modelling
In this deep learning project, you will find similar images (lookalikes) using deep learning and locality sensitive hashing to find customers who are most likely to click on an ad.

Natural language processing Chatbot application using NLTK for text classification
In this NLP AI application, we build the core conversational engine for a chatbot. We use the popular NLTK text classification library to achieve this.

Topic modelling using Kmeans clustering to group customer reviews
In this Kmeans clustering machine learning project, you will perform topic modelling in order to group customer reviews based on recurring patterns.

Demand prediction of driver availability using multistep time series analysis
In this supervised learning machine learning project, you will predict the availability of a driver in a specific area by using multi step time series analysis.

House Price Prediction Project using Machine Learning
Use the Zillow dataset to follow a test-driven approach and build a regression machine learning model to predict the price of the house based on other variables.

Churn Prediction in Telecom using Machine Learning in R
Estimating churners before they discontinue using a product or service is extremely important. In this ML project, you will develop a churn prediction model in telecom to predict customers who are most likely subject to churn.

Expedia Hotel Recommendations Data Science Project
In this data science project, you will contextualize customer data and predict the likelihood a customer will stay at 100 different hotel groups.

Medical Image Segmentation Deep Learning Project
In this deep learning project, you will learn to implement Unet++ models for medical image segmentation to detect and classify colorectal polyps.