What are Retrieval based models in transformers?

Recipe Objective - What are Retrieval-based models in transformers?

For open-domain question answering, retrieval-based models leverage documents retrieval during (pre)training and inference.

Types of Retrieval-based model:
DPR:

Dense Passage Retrieval (DPR) is a set of tools and models for open-domain question-answering research at the cutting edge.

DPR is divided into three types:
1. Question encoder: encode questions as vectors
2. Context encoder: encode contexts as vectors
3. Reader: extract the questions' answers, as well as a relevance score, from the retrieved contexts (high if the inferred span actually answers the question).

The DPR pipeline (which has yet to be implemented) employs a retrieval phase to discover the top k contexts for a given question, and then it calls the reader with the query and the retrieved documents to receive the response.

RAG:

Pretrained dense retrieval (DPR) and Seq2Seq models are combined in retrieval-augmented generation (RAG) models. RAG models retrieve documents, feed them to a seq2seq model, and minimise the results to produce outputs. The retriever and seq2seq modules are both fine-tuned together after being initialised from pre-trained models, allowing retrieval and generation to adapt to downstream requirements.

For more related projects -

/projects/data-science-projects/tensorflow-projects
/projects/data-science-projects/keras-deep-learning-projects