What is Masked Language Modeling in transformers?

This recipe explains what is Masked Language Modeling in transformers.

Recipe Objective - What is Masked Language Modeling in transformers?

The task of fitting a model to a corpus, which can be domain-specific, is known as language modeling. Language modeling versions, such as BERT with masked language modeling and GPT2 with causal language modeling, are used to train all popular transformers-based models.

Language modeling is also useful outside of pre-training, for example, to transform the model distribution in a specific domain: use a trained language model on a very large corpus and then fit it to data sets from news or scientific articles, such as LysandreJik / arxivnlp.

Learn How to Build a Multi Class Text Classification Model using BERT

Masked Language Modeling:

The task of masking tokens in a sequence with a masking token and directing the model to fill that mask with an appropriate token is known as masked language modeling. This allows the model to focus on both the right and left contexts (tokens on the right side of the mask) (tokens on the left of the mask).

For more related projects -

/projects/data-science-projects/deep-learning-projects
/projects/data-science-projects/neural-network-projects

Example of Masked Language Model using pipeline:

# Importing libraries
from transformers import pipeline
from pprint import pprint

# Creating model using pipeline for question answering
Model_unmasker = pipeline("fill-mask")

# Defining content for question
pprint(Model_unmasker(f"I have never watched something like this {unmasker.tokenizer.mask_token} and it was amazing."))

Output -
[{'score': 0.8949385285377502,
  'sequence': 'I have never watched something like this before and it was '
              'amazing.',
  'token': 137,
  'token_str': ' before'},
 {'score': 0.01026323065161705,
  'sequence': 'I have never watched something like this movie and it was '
              'amazing.',
  'token': 1569,
  'token_str': ' movie'},
 {'score': 0.008902600035071373,
  'sequence': 'I have never watched something like this, and it was amazing.',
  'token': 6,
  'token_str': ','},
 {'score': 0.007995985448360443,
  'sequence': 'I have never watched something like this again and it was '
              'amazing.',
  'token': 456,
  'token_str': ' again'},
 {'score': 0.0057492610067129135,
  'sequence': 'I have never watched something like this film and it was '
              'amazing.',
  'token': 822,
  'token_str': ' film'}]

In this way, we can perform masked language modeling in transformers.

What Users are saying..

profile image

Gautam Vermani

Data Consultant at Confidential
linkedin profile url

Having worked in the field of Data Science, I wanted to explore how I can implement projects in other domains, So I thought of connecting with ProjectPro. A project that helped me absorb this topic... Read More

Relevant Projects

Learn Hyperparameter Tuning for Neural Networks with PyTorch
In this Deep Learning Project, you will learn how to optimally tune the hyperparameters (learning rate, epochs, dropout, early stopping) of a neural network model in PyTorch to improve model performance.

Learn to Build an End-to-End Machine Learning Pipeline - Part 1
In this Machine Learning Project, you will learn how to build an end-to-end machine learning pipeline for predicting truck delays, addressing a major challenge in the logistics industry.

Topic modelling using Kmeans clustering to group customer reviews
In this Kmeans clustering machine learning project, you will perform topic modelling in order to group customer reviews based on recurring patterns.

MLOps using Azure Devops to Deploy a Classification Model
In this MLOps Azure project, you will learn how to deploy a classification machine learning model to predict the customer's license status on Azure through scalable CI/CD ML pipelines.

Hands-On Approach to Causal Inference in Machine Learning
In this Machine Learning Project, you will learn to implement various causal inference techniques in Python to determine, how effective the sprinkler is in making the grass wet.

Detectron2 Object Detection and Segmentation Example Python
Object Detection using Detectron2 - Build a Dectectron2 model to detect the zones and inhibitions in antibiogram images.

Build a CNN Model with PyTorch for Image Classification
In this deep learning project, you will learn how to build an Image Classification Model using PyTorch CNN

Deploy Transformer-BART Model on Paperspace Cloud
In this MLOps Project you will learn how to deploy a Tranaformer BART Model for Abstractive Text Summarization on Paperspace Private Cloud

Multi-Class Text Classification with Deep Learning using BERT
In this deep learning project, you will implement one of the most popular state of the art Transformer models, BERT for Multi-Class Text Classification

PyTorch Project to Build a LSTM Text Classification Model
In this PyTorch Project you will learn how to build an LSTM Text Classification model for Classifying the Reviews of an App .