# Retrieving whole words with fill-mask pipeline

**URL:** <https://discuss.huggingface.co/t/retrieving-whole-words-with-fill-mask-pipeline/11323>\
**Category:** Beginners\
**Created:** [November 3, 2021, 1:40pm UTC](https://discuss.huggingface.co/t/retrieving-whole-words-with-fill-mask-pipeline/11323 "2021-11-03T13:40:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![fmorales](https://avatars.discourse-cdn.com/v4/letter/f/da6949/32.png) [@fmorales](https://discuss.huggingface.co/u/fmorales)\
**Post date:** [November 3, 2021, 1:40pm UTC](https://discuss.huggingface.co/t/retrieving-whole-words-with-fill-mask-pipeline/11323/1 "2021-11-03T13:40:24Z")

</div>

Hi,

I’ve recently discovered the power of the fill-mask pipeline from Huggingface, and while playing with it, I discovered that it has issues handling non-vocabulary words.

For example, in the sentence, “The internal analysis indicates that the company has reached a [MASK] level.”, I would like to know which one of these words [‘good’, ‘virtuous’, ‘obedient’] is the most probable according to the bert-large-cased-whole-word-masking model.

The model refuses to give a score to the words virtuous and obedient because they do not exist in the vocabulary as such, therefore the scores are given to the first tokens that are recognized: v and o; which are not useful.

So the question remains, how could I get the prediction scores for the whole word instead of scores for individual subword tokens?

---

<div class="post-metadata">

**Author:** ![Felipehonorato](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/felipehonorato/32/48418_2.png) [@Felipehonorato](https://discuss.huggingface.co/u/Felipehonorato)\
**Post date:** [November 19, 2021, 7:05pm UTC](https://discuss.huggingface.co/t/retrieving-whole-words-with-fill-mask-pipeline/11323/2 "2021-11-19T19:05:19Z")

</div>

I’m not sure but you would average the probability of these tokens together and then compare with each other