How does BertTokenizer work in transformers?
This recipe explains how does BertTokenizer work in transformers.
Recipe Objective - How does BertTokenizer work in transformers?
Subword tokenization methods work on the idea that common words should not be broken down into smaller subwords, but rare words should be broken down into meaningful subwords.
Access Avocado Machine Learning Project for Price Prediction
For more related projects -
/projects/data-science-projects/neural-network-projects
/projects/data-science-projects/tensorflow-projects
Example of BertTokenizer:
# Importing BertTokenizer
from transformers import BertTokenizer
bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
# Passing input
bert_tokenizer.tokenize("Welcome to Transformers tutorials!!!")
Output - ['welcome', 'to', 'transformers', 'tutor', '##ials', '!', '!', '!']
The sentence was lowercased first because we're using the uncased model. We can see that the words ["welcome", "to", "transformers"] are present in the tokenizer’s vocabulary, but the word "tutorials" is not. Consequently, the tokenizer splits "tutorials" into known subwords: ["tutor" and "##ials"]. The symbol "##" indicates that the remainder of the token should be connected to the previous one without leaving any gap (for decoding or reversal of the tokenization).
In this way, we can perform BertTokenizer in transformers.
What Users are saying..
Abhinav Agarwal
I come from Northwestern University, which is ranked 9th in the US. Although the high-quality academics at school taught me all the basics I needed, obtaining practical experience was a challenge.... Read More