What are multimodal models in transformers?
This recipe explains what are multimodal models in transformers.
Recipe Objective - What are multimodal models in transformers?
Multimodal models combine text and other types of input (such as graphics, images etc.) and are more task-specific. One multimodal model in the collection has not been pre-trained in the same self-supervised manner as the others.
Build a Multi Touch Attribution Model in Python with Source Code
Type of multimodal model:
MMBT:
In multimodal environments, a transformers model is used to create predictions by merging text and image. The embeddings of the tokenized text and the final activations of a pretrained on pictures resnet (after the pooling layer) that travels via a linear layer are fed into the transformer model (to go from number of features at the end of the resnet to the hidden state dimension of the transformer). The different inputs are combined, and a segment embedding is added on top of the positional embeddings to tell the model which part of the input vector relates to the text and which to the image. Only classification is possible with the pretrained model.
For more related projects -
/projects/data-science-projects/deep-learning-projects
/projects/data-science-projects/keras-deep-learning-projects
What Users are saying..
Abhinav Agarwal
I come from Northwestern University, which is ranked 9th in the US. Although the high-quality academics at school taught me all the basics I needed, obtaining practical experience was a challenge.... Read More