How to find n grams in pytorch

This recipe helps you find n grams in pytorch

Recipe Objective

How to find n_grams in Pytorch?

This is achieved by using "ngrams_iterator" which will return the iterator that yields the given tokens and their ngrams. Firstly we have take sample text then we have to create tokens by tokenizing the text, and from tokens will create token_list for ngrams.

PyTorch vs Tensorflow - Which One Should You Choose For Your Next Deep Learning Project ?

Step 1 - Import library

import torchtext
from torchtext.data
import get_tokenizer
from torchtext.data.utils
import ngrams_iterator

Step 2 - Take Sample text

text = "This is a pytorch tutorial for ngrams"

Step 3 - Create tokens

torch_tokenizer = get_tokenizer("spacy")
tokens = torch_tokenizer(text)
print("The tokens for Sample text:",tokens)

The tokens for Sample text: ['This', 'is', 'a', 'pytorch', 'tutorial', 'for', 'ngrams']

Step 4 - Create token list

token_list = list(tokens)
print("This is the list of tokens",token_list)

This is the list of tokens ['This', 'is', 'a', 'pytorch', 'tutorial', 'for', 'ngrams']

Step 5 - Create ngrams

ngrams = list(ngrams_iterator(token_list, 2))
print("The ngrams for token list is:",ngrams)

The ngrams for token list is: ['This', 'is', 'a', 'pytorch', 'tutorial', 'for', 'ngrams', 'This is', 'is a', 'a pytorch', 'pytorch tutorial', 'tutorial for', 'for ngrams']

What Users are saying..

profile image

Ed Godalle

Director Data Analytics at EY / EY Tech
linkedin profile url

I am the Director of Data Analytics with over 10+ years of IT experience. I have a background in SQL, Python, and Big Data working with Accenture, IBM, and Infosys. I am looking to enhance my skills... Read More

Relevant Projects

Build a Multi-Class Classification Model in Python on Saturn Cloud
In this machine learning classification project, you will build a multi-class classification model in Python on Saturn Cloud to predict the license status of a business.

Recommender System Machine Learning Project for Beginners-1
Recommender System Machine Learning Project for Beginners - Learn how to design, implement and train a rule-based recommender system in Python

Build a Logistic Regression Model in Python from Scratch
Regression project to implement logistic regression in python from scratch on streaming app data.

Machine Learning Project to Forecast Rossmann Store Sales
In this machine learning project you will work on creating a robust prediction model of Rossmann's daily sales using store, promotion, and competitor data.

Predict Churn for a Telecom company using Logistic Regression
Machine Learning Project in R- Predict the customer churn of telecom sector and find out the key drivers that lead to churn. Learn how the logistic regression model using R can be used to identify the customer churn in telecom dataset.

Multi-Class Text Classification with Deep Learning using BERT
In this deep learning project, you will implement one of the most popular state of the art Transformer models, BERT for Multi-Class Text Classification

Classification Projects on Machine Learning for Beginners - 2
Learn to implement various ensemble techniques to predict license status for a given business.

Build a Customer Churn Prediction Model using Decision Trees
Develop a customer churn prediction model using decision tree machine learning algorithms and data science on streaming service data.

Loan Eligibility Prediction Project using Machine learning on GCP
Loan Eligibility Prediction Project - Use SQL and Python to build a predictive model on GCP to determine whether an application requesting loan is eligible or not.

Build a Speech-Text Transcriptor with Nvidia Quartznet Model
In this Deep Learning Project, you will leverage transfer learning from Nvidia QuartzNet pre-trained models to develop a speech-to-text transcriptor.