How to find phrase and sentence similarity using Skip Gram model?
As we have discussed earlier Skip Gram, so it predicts the surrounding context words within a specific window given the current word. The input layer contains the current word and the output layer contains the setting words. The layer which is hidden contains the number of measurements wherein we need to do the representation of the current word present at the input layer.
Sentence and phrase similarity So these are terms in which we are going to detect the similarity between two sentences or phrases. In the event that equivalent words exist in both the sentences or expressions in the same request then we can say there is a sentence or phrase similarity, yet in the event that the words are in various order, at that point it should not completely consider a similar sentence or phrase. Let's understand this with practical implementation.
from nltk.tokenize import sent_tokenize, word_tokenize
warnings.filterwarnings(action = 'ignore')
from gensim.models import Word2Vec
Here we have imported the necessary packages along with the warnings and kept it as ignore because we know that there might be some warnings comming up when we run our program, but that can be ignored.
Sample = open("/content/alice_in_wonderland.txt","r")
s = Sample.read()
f = s.replace("\n", " ")
[nltk_data] Downloading package punkt to /root/nltk_data... [nltk_data] Unzipping tokenizers/punkt.zip.
data = 
for i in sent_tokenize(f):
temp = 
for j in word_tokenize(i):
Here we are taking a list as variable named data which is initially empty, after that we are going take a for loop which will iterate through each sentences present in the text file, and the second for loop will tokenize the sentences into words.
model2 = gensim.models.Word2Vec(data, min_count = 1, size = 100, window = 5, sg = 1)
print("Similarity between 'alice' " + "and 'girl' - Skip Gram : ", model2.similarity('alice', 'girl'),'\n')
print("Similarity between 'alice' " + "and 'boy' - Skip Gram : ", model2.similarity('alice', 'boy'), '\n')
print("Similarity between 'alice' " + "and 'rabbit' - Skip Gram : ",
Similarity between 'alice' and 'girl' - Skip Gram : 0.951415 Similarity between 'alice' and 'boy' - Skip Gram : 0.94738436 Similarity between 'alice' and 'rabbit' - Skip Gram : 0.97160363 Similarity between 'girl' and 'rabbit' - Skip Gram : 0.98851126 Similarity between 'girl' and 'cat' - Skip Gram : 0.9877583
Here from the above we can see the similarity between the words, we have used the words which are repeatedly used in the Sample text. So we can see the similarity between them by using skip gram model.