What is Masked Language Modeling in transformers?

This recipe explains what is Masked Language Modeling in transformers.

Recipe Objective - What is Masked Language Modeling in transformers?

The task of fitting a model to a corpus, which can be domain-specific, is known as language modeling. Language modeling versions, such as BERT with masked language modeling and GPT2 with causal language modeling, are used to train all popular transformers-based models.

Language modeling is also useful outside of pre-training, for example, to transform the model distribution in a specific domain: use a trained language model on a very large corpus and then fit it to data sets from news or scientific articles, such as LysandreJik / arxivnlp.

Learn How to Build a Multi Class Text Classification Model using BERT

Masked Language Modeling:

The task of masking tokens in a sequence with a masking token and directing the model to fill that mask with an appropriate token is known as masked language modeling. This allows the model to focus on both the right and left contexts (tokens on the right side of the mask) (tokens on the left of the mask).

For more related projects -

/projects/data-science-projects/deep-learning-projects
/projects/data-science-projects/neural-network-projects

Example of Masked Language Model using pipeline:

# Importing libraries
from transformers import pipeline
from pprint import pprint

# Creating model using pipeline for question answering
Model_unmasker = pipeline("fill-mask")

# Defining content for question
pprint(Model_unmasker(f"I have never watched something like this {unmasker.tokenizer.mask_token} and it was amazing."))

Output -
[{'score': 0.8949385285377502,
  'sequence': 'I have never watched something like this before and it was '
              'amazing.',
  'token': 137,
  'token_str': ' before'},
 {'score': 0.01026323065161705,
  'sequence': 'I have never watched something like this movie and it was '
              'amazing.',
  'token': 1569,
  'token_str': ' movie'},
 {'score': 0.008902600035071373,
  'sequence': 'I have never watched something like this, and it was amazing.',
  'token': 6,
  'token_str': ','},
 {'score': 0.007995985448360443,
  'sequence': 'I have never watched something like this again and it was '
              'amazing.',
  'token': 456,
  'token_str': ' again'},
 {'score': 0.0057492610067129135,
  'sequence': 'I have never watched something like this film and it was '
              'amazing.',
  'token': 822,
  'token_str': ' film'}]

In this way, we can perform masked language modeling in transformers.

What Users are saying..

profile image

Abhinav Agarwal

Graduate Student at Northwestern University
linkedin profile url

I come from Northwestern University, which is ranked 9th in the US. Although the high-quality academics at school taught me all the basics I needed, obtaining practical experience was a challenge.... Read More

Relevant Projects

Create Your First Chatbot with RASA NLU Model and Python
Learn the basic aspects of chatbot development and open source conversational AI RASA to create a simple AI powered chatbot on your own.

Loan Eligibility Prediction in Python using H2O.ai
In this loan prediction project you will build predictive models in Python using H2O.ai to predict if an applicant is able to repay the loan or not.

Build a Credit Default Risk Prediction Model with LightGBM
In this Machine Learning Project, you will build a classification model for default prediction with LightGBM.

Learn to Build a Siamese Neural Network for Image Similarity
In this Deep Learning Project, you will learn how to build a siamese neural network with Keras and Tensorflow for Image Similarity.

Build an optimal End-to-End MLOps Pipeline and Deploy on GCP
Learn how to build and deploy an end-to-end optimal MLOps Pipeline for Loan Eligibility Prediction Model in Python on GCP

Isolation Forest Model and LOF for Anomaly Detection in Python
Credit Card Fraud Detection Project - Build an Isolation Forest Model and Local Outlier Factor (LOF) in Python to identify fraudulent credit card transactions.

Deploy Transformer-BART Model on Paperspace Cloud
In this MLOps Project you will learn how to deploy a Tranaformer BART Model for Abstractive Text Summarization on Paperspace Private Cloud

Recommender System Machine Learning Project for Beginners-4
Collaborative Filtering Recommender System Project - Comparison of different model based and memory based methods to build recommendation system using collaborative filtering.

MLOps Project for a Mask R-CNN on GCP using uWSGI Flask
MLOps on GCP - Solved end-to-end MLOps Project to deploy a Mask RCNN Model for Image Segmentation as a Web Application using uWSGI Flask, Docker, and TensorFlow.

GCP MLOps Project to Deploy ARIMA Model using uWSGI Flask
Build an end-to-end MLOps Pipeline to deploy a Time Series ARIMA Model on GCP using uWSGI and Flask