How to find phrase and sentence similarity using Skip Gram model in nlp

This recipe helps you find phrase and sentence similarity using Skip Gram model in nlp

Recipe Objective

How to find phrase and sentence similarity using Skip Gram model?

As we have discussed earlier Skip Gram, so it predicts the surrounding context words within a specific window given the current word. The input layer contains the current word and the output layer contains the setting words. The layer which is hidden contains the number of measurements wherein we need to do the representation of the current word present at the input layer.

Sentence and phrase similarity So these are terms in which we are going to detect the similarity between two sentences or phrases. In the event that equivalent words exist in both the sentences or expressions in the same request then we can say there is a sentence or phrase similarity, yet in the event that the words are in various order, at that point it should not completely consider a similar sentence or phrase. Let's understand this with practical implementation.

Step 1 - Import the necessary libraries

from nltk.tokenize import sent_tokenize, word_tokenize import warnings warnings.filterwarnings(action = 'ignore') import gensim from gensim.models import Word2Vec

Here we have imported the necessary packages along with the warnings and kept it as ignore because we know that there might be some warnings comming up when we run our program, but that can be ignored.

Step 2 - load the sample data

Sample = open("/content/alice_in_wonderland.txt","r") s = Sample.read()

Step 3 - Replace the escape character with spaces

f = s.replace("\n", " ")

Step 4 - Iterate and tokenize

import nltk nltk.download('punkt')

[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
True

data = [] for i in sent_tokenize(f): temp = [] for j in word_tokenize(i): temp.append(j.lower()) data.append(temp)

Here we are taking a list as variable named data which is initially empty, after that we are going take a for loop which will iterate through each sentences present in the text file, and the second for loop will tokenize the sentences into words.

Step 5 - Create a Skip Gram model

model2 = gensim.models.Word2Vec(data, min_count = 1, size = 100, window = 5, sg = 1)

Step 6 - Print the result of Skip Gram model

print("Similarity between 'alice' " + "and 'girl' - Skip Gram : ", model2.similarity('alice', 'girl'),'\n') print("Similarity between 'alice' " + "and 'boy' - Skip Gram : ", model2.similarity('alice', 'boy'), '\n') print("Similarity between 'alice' " + "and 'rabbit' - Skip Gram : ",

Similarity between 'alice' and 'girl' - Skip Gram :  0.951415 

Similarity between 'alice' and 'boy' - Skip Gram :  0.94738436 

Similarity between 'alice' and 'rabbit' - Skip Gram :  0.97160363 

Similarity between 'girl' and 'rabbit' - Skip Gram :  0.98851126 

Similarity between 'girl' and 'cat' - Skip Gram :  0.9877583

Here from the above we can see the similarity between the words, we have used the words which are repeatedly used in the Sample text. So we can see the similarity between them by using skip gram model.

What Users are saying..

profile image

Anand Kumpatla

Sr Data Scientist @ Doubleslash Software Solutions Pvt Ltd
linkedin profile url

ProjectPro is a unique platform and helps many people in the industry to solve real-life problems with a step-by-step walkthrough of projects. A platform with some fantastic resources to gain... Read More

Relevant Projects

Build a Text Classification Model with Attention Mechanism NLP
In this NLP Project, you will learn to build a multi class text classification model with attention mechanism.

Recommender System Machine Learning Project for Beginners-4
Collaborative Filtering Recommender System Project - Comparison of different model based and memory based methods to build recommendation system using collaborative filtering.

Build a Speech-Text Transcriptor with Nvidia Quartznet Model
In this Deep Learning Project, you will leverage transfer learning from Nvidia QuartzNet pre-trained models to develop a speech-to-text transcriptor.

Deep Learning Project for Text Detection in Images using Python
CV2 Text Detection Code for Images using Python -Build a CRNN deep learning model to predict the single-line text in a given image.

Build a Customer Churn Prediction Model using Decision Trees
Develop a customer churn prediction model using decision tree machine learning algorithms and data science on streaming service data.

Expedia Hotel Recommendations Data Science Project
In this data science project, you will contextualize customer data and predict the likelihood a customer will stay at 100 different hotel groups.

Recommender System Machine Learning Project for Beginners-3
Content Based Recommender System Project - Building a Content-Based Product Recommender App with Streamlit

Build a Collaborative Filtering Recommender System in Python
Use the Amazon Reviews/Ratings dataset of 2 Million records to build a recommender system using memory-based collaborative filtering in Python.

Learn to Build a Siamese Neural Network for Image Similarity
In this Deep Learning Project, you will learn how to build a siamese neural network with Keras and Tensorflow for Image Similarity.

Text Classification with Transformers-RoBERTa and XLNet Model
In this machine learning project, you will learn how to load, fine tune and evaluate various transformer models for text classification tasks.