What are Decoders or autoregressive models in transformers?

This recipe explains what are Decoders or autoregressive models in transformers.
Last Updated: 08 Aug 2022

Get access to Data Science projects View all Data Science projects

MACHINE LEARNING RECIPES DATA CLEANING PYTHON DATA MUNGING PANDAS CHEATSHEET ALL TAGS

Recipe Objective - What are Decoders or autoregressive models in transformers?

Decoders, also known as autoregressive models, are trained on the traditional language modelling problem of guessing the next token after reading the preceding ones. They correspond to the original transformer model's decoder, and a mask is applied to the entire phrase so that the attention heads can only perceive what came before in the text, not what comes after. Although these models can be fine-tuned to produce excellent outcomes for a variety of tasks, text production is the most natural use. The GPT model is a good example of this type of paradigm.

Access Avocado Machine Learning Project for Price Prediction

Types of Decoders or autoregressive models:

* Original GPT
* GPT-2
* CTRL
* Transformer-XL
* Reformer
* XLNet

For more related projects -

/projects/data-science-projects/tensorflow-projects
/projects/data-science-projects/keras-deep-learning-projects

What Users are saying..

Gautam Vermani

Data Consultant at Confidential

Having worked in the field of Data Science, I wanted to explore how I can implement projects in other domains, So I thought of connecting with ProjectPro. A project that helped me absorb this topic... Read More

Relevant Projects

Machine Learning Projects

Data Science Projects

Python Projects for Data Science

Data Science Projects in R

Machine Learning Projects for Beginners

Deep Learning Projects

Neural Network Projects

Tensorflow Projects

NLP Projects

Kaggle Projects

IoT Projects

Big Data Projects

Hadoop Real-Time Projects Examples

Spark Projects

Data Analytics Projects for Students

Relevant Projects

Many-to-One LSTM for Sentiment Analysis and Text Generation

In this LSTM Project , you will build develop a sentiment detection model using many-to-one LSTMs for accurate prediction of sentiment labels in airline text reviews. Additionally, we will also train many-to-one LSTMs on 'Alice's Adventures in Wonderland' to generate contextually relevant text.

View Project Details

Learn How to Build a Logistic Regression Model in PyTorch

In this Machine Learning Project, you will learn how to build a simple logistic regression model in PyTorch for customer churn prediction.

View Project Details

Build Multi Class Text Classification Models with RNN and LSTM

In this Deep Learning Project, you will use the customer complaints data about consumer financial products to build multi-class text classification models using RNN and LSTM.

View Project Details

Deep Learning Project- Real-Time Fruit Detection using YOLOv4

In this deep learning project, you will learn to build an accurate, fast, and reliable real-time fruit detection system using the YOLOv4 object detection model for robotic harvesting platforms.

View Project Details

Expedia Hotel Recommendations Data Science Project

In this data science project, you will contextualize customer data and predict the likelihood a customer will stay at 100 different hotel groups.

View Project Details

Build Piecewise and Spline Regression Models in Python

In this Regression Project, you will learn how to build a piecewise and spline regression model from scratch in Python to predict the points scored by a sports team.

View Project Details