What is a model in Gensim

Recipe Objective: What is a model in Gensim?

Once our corpus has been vectorized, we can change it using models. We use the term "model" to refer to a change from one representation of a document to another. Because documents in gensim are represented as vectors, a model can be considered a transition between two vector spaces. When the model reads the training Corpus during training, it learns the details of this change.

tf-idf is a simple example of a model. The tf-idf model converts vectors from a bag-of-words representation to a vector space where frequency counts are weighted according to each word's relative rarity in the corpus.

Here's a simple illustration. Let's get started by training the tf-idf model on our corpus and translating the string "sample corpus":

Build a Chatbot in Python from Scratch!

#importing required libraries
from gensim import models
import gensim
from gensim import corpora

#creating a sample corpus for demonstration purpose
txt_corpus = ["This is sample document",
"Collection of documents make a corpus",
"You can vectorize your corpus"]

# Creating a set of frequent words
stoplist = set('for a of the and to in on of to are at'.split(' '))

# Lowercasing each document, using white space as delimiter and filtering out the stopwords
processed_text = [[word for word in document.lower().split() if word not in stoplist]for document in txt_corpus]

#creating a dictionary
dictionary = corpora.Dictionary(processed_text)

#using doc2bow for vectorization of the entire corpus
bow_vec = [dictionary.doc2bow(text) for text in processed_text]

#training the model
tfidf_model = models.TfidfModel(bow_vec)

#transforming the "sample corpus" string
words = "sample corpus".lower().split()
print(tfidf_model[dictionary.doc2bow(words)])


Output:
[(2, 0.9381453975456102), (5, 0.34624155305796134)]

The tf-idf model returns a list of tuples, with the token ID as the first element and the tf-idf weighting as the second. It's worth noting that the ID for "corpus"(which appeared two times in the original corpus) has been weighted lower than the ID for "sample" (which only occurred once).