What are Retrieval based models in transformers?

This recipe explains what are Retrieval based models in transformers.

Recipe Objective - What are Retrieval-based models in transformers?

For open-domain question answering, retrieval-based models leverage documents retrieval during (pre)training and inference.

Types of Retrieval-based model:
DPR:

Dense Passage Retrieval (DPR) is a set of tools and models for open-domain question-answering research at the cutting edge.

DPR is divided into three types:
1. Question encoder: encode questions as vectors
2. Context encoder: encode contexts as vectors
3. Reader: extract the questions' answers, as well as a relevance score, from the retrieved contexts (high if the inferred span actually answers the question).

The DPR pipeline (which has yet to be implemented) employs a retrieval phase to discover the top k contexts for a given question, and then it calls the reader with the query and the retrieved documents to receive the response.

RAG:

Pretrained dense retrieval (DPR) and Seq2Seq models are combined in retrieval-augmented generation (RAG) models. RAG models retrieve documents, feed them to a seq2seq model, and minimise the results to produce outputs. The retriever and seq2seq modules are both fine-tuned together after being initialised from pre-trained models, allowing retrieval and generation to adapt to downstream requirements.

For more related projects -

/projects/data-science-projects/tensorflow-projects
/projects/data-science-projects/keras-deep-learning-projects

What Users are saying..

profile image

Abhinav Agarwal

Graduate Student at Northwestern University
linkedin profile url

I come from Northwestern University, which is ranked 9th in the US. Although the high-quality academics at school taught me all the basics I needed, obtaining practical experience was a challenge.... Read More

Relevant Projects

Deep Learning Project for Time Series Forecasting in Python
Deep Learning for Time Series Forecasting in Python -A Hands-On Approach to Build Deep Learning Models (MLP, CNN, LSTM, and a Hybrid Model CNN-LSTM) on Time Series Data.

Demand prediction of driver availability using multistep time series analysis
In this supervised learning machine learning project, you will predict the availability of a driver in a specific area by using multi step time series analysis.

Personalized Medicine: Redefining Cancer Treatment
In this Personalized Medicine Machine Learning Project you will learn to classify genetic mutations on the basis of medical literature into 9 classes.

Learn Hyperparameter Tuning for Neural Networks with PyTorch
In this Deep Learning Project, you will learn how to optimally tune the hyperparameters (learning rate, epochs, dropout, early stopping) of a neural network model in PyTorch to improve model performance.

Learn to Build an End-to-End Machine Learning Pipeline - Part 1
In this Machine Learning Project, you will learn how to build an end-to-end machine learning pipeline for predicting truck delays, addressing a major challenge in the logistics industry.

PyTorch Project to Build a LSTM Text Classification Model
In this PyTorch Project you will learn how to build an LSTM Text Classification model for Classifying the Reviews of an App .

Locality Sensitive Hashing Python Code for Look-Alike Modelling
In this deep learning project, you will find similar images (lookalikes) using deep learning and locality sensitive hashing to find customers who are most likely to click on an ad.

Image Segmentation using Mask R-CNN with Tensorflow
In this Deep Learning Project on Image Segmentation Python, you will learn how to implement the Mask R-CNN model for early fire detection.

Build a Music Recommendation Algorithm using KKBox's Dataset
Music Recommendation Project using Machine Learning - Use the KKBox dataset to predict the chances of a user listening to a song again after their very first noticeable listening event.

Build a Review Classification Model using Gated Recurrent Unit
In this Machine Learning project, you will build a classification model in python to classify the reviews of an app on a scale of 1 to 5 using Gated Recurrent Unit.