# Multiple Mask Tokens

**URL:** <https://discuss.huggingface.co/t/multiple-mask-tokens/174>\
**Category:** 🤗Transformers\
**Created:** [July 10, 2020, 12:34pm UTC](https://discuss.huggingface.co/t/multiple-mask-tokens/174 "2020-07-10T12:34:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zanderbush](https://avatars.discourse-cdn.com/v4/letter/z/13edae/32.png) [@zanderbush](https://discuss.huggingface.co/u/zanderbush)\
**Post date:** [July 10, 2020, 12:34pm UTC](https://discuss.huggingface.co/t/multiple-mask-tokens/174/1 "2020-07-10T12:34:40Z")

</div>

For those wishing to [MASK] several tokens, here this is.

My question, however, relates to the output. I added “top\_k” assuming I’d be able to return multiple sentences, but that was not the case. I am not sure how exactly I can achieve this.

```
import torch
from transformers import BertTokenizer, BertModel,BertForMaskedLM
 
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
 
input_tx = "[CLS] [MASK] [MASK] [MASK] of the United States mismangement of the Coronavirus is its distrust of science. [SEP]"
tokenized_text = tokenizer.tokenize(input_tx)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
top_k = 10
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([[0]*25])
 
model = BertForMaskedLM.from_pretrained('bert-base-cased')
 
outputs = model(tokens_tensor, token_type_ids=segments_tensors)
predictions = outputs[0]
predicted_index = [torch.argmax(predictions[0, i]).item() for i in range(0,24)]
predicted_token = [tokenizer.convert_ids_to_tokens([predicted_index[x]])[0] for x in range(1,24)]
print(predicted_token)

`Output: 'The', 'main', 'cause', 'of', 'the', 'United', 'States', 'mi', '##sman', '##gement', 'of', 'the', 'Co', '##rona', '##virus', 'is', 'its', 'di', '##st', '##rust', 'of', 'science', '`
```

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [July 10, 2020, 1:51pm UTC](https://discuss.huggingface.co/t/multiple-mask-tokens/174/2 "2020-07-10T13:51:41Z")

</div>

Hi there! First of all, please note that in the latest release, the recommended way to preprocess your input is just to call the tokenizer on your test:

```python
import torch
from transformers import BertTokenizer, BertModel,BertForMaskedLM
 
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
input_txt = "[MASK] [MASK] [MASK] of the United States mismanagement of the Coronavirus is its distrust of science."
inputs = tokenizer(input_txt, return_tensors='pt')

```

This returns a dict string to tensors (since I asked to return pytorch tensors with the last argument) and you can then directly call your model on it:

```auto
model = BertForMaskedLM.from_pretrained('bert-base-cased')
 
outputs = model(**inputs)
predictions = outputs[0]

```

At this stage, `predictions` is the output of our language model before the softmax (we won’t care about that since the probabilities after the softmax or the activations before are in the same order). You ask for the most probable token, so it only returns that. If you want, say, the most probable 10 tokens, you could go:

```python
sorted_preds, sorted_idx = predictions[0].sort(dim=-1, descending=True)
for k in range(10):
    predicted_index = [sorted_idx[i, k].item() for i in range(0,24)]
    predicted_token = [tokenizer.convert_ids_to_tokens([predicted_index[x]])[0] for x in range(1,24)]
    print(predicted_token)

```

---

<div class="post-metadata">

**Author:** ![zanderbush](https://avatars.discourse-cdn.com/v4/letter/z/13edae/32.png) [@zanderbush](https://discuss.huggingface.co/u/zanderbush)\
**Post date:** [July 10, 2020, 3:50pm UTC](https://discuss.huggingface.co/t/multiple-mask-tokens/174/3 "2020-07-10T15:50:36Z")

</div>

Thank you so much for your help. I really appreciate your in-depth explanation.

---

<div class="post-metadata">

**Author:** ![neerajak](https://avatars.discourse-cdn.com/v4/letter/n/6bbea6/32.png) [@neerajak](https://discuss.huggingface.co/u/neerajak)\
**Post date:** [July 18, 2021, 10:28am UTC](https://discuss.huggingface.co/t/multiple-mask-tokens/174/4 "2021-07-18T10:28:38Z")

</div>

Is there a way to retrieve the probabilities of the words retrieved in the multiple masks? Any help will be really appreciated.

---

<div class="post-metadata">

**Author:** ![breandan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/breandan/32/4009_2.png) [@breandan](https://discuss.huggingface.co/u/breandan)\
**Post date:** [February 12, 2022, 11:17pm UTC](https://discuss.huggingface.co/t/multiple-mask-tokens/174/5 "2022-02-12T23:17:59Z")

</div>

> Is there a way to retrieve the probabilities of the words retrieved in the multiple masks?

Since experimental support for multi-masking was recently added to the `fill-mask` pipeline, retrieving the `score` for each token is supported, although it is still unclear clear what the correct semantics should be. If you want to use the `targets` parameter, this is only supported in the single-mask case. @Lysandre goes into further details in the [PR discussion](https://github.com/huggingface/transformers/pull/10222#issuecomment-789816310).