# How does FillMaskPipeline work with Subword-Tokenization?

**URL:** <https://discuss.huggingface.co/t/how-does-fillmaskpipeline-work-with-subword-tokenization/15820>\
**Category:** 🤗Transformers\
**Created:** [March 17, 2022, 6:43pm UTC](https://discuss.huggingface.co/t/how-does-fillmaskpipeline-work-with-subword-tokenization/15820 "2022-03-17T18:43:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![harnisph](https://avatars.discourse-cdn.com/v4/letter/h/f0a364/32.png) [@harnisph](https://discuss.huggingface.co/u/harnisph)\
**Post date:** [March 17, 2022, 6:43pm UTC](https://discuss.huggingface.co/t/how-does-fillmaskpipeline-work-with-subword-tokenization/15820/1 "2022-03-17T18:43:12Z")

</div>

I have a general understanding problem in regard to the FillMaskPipeline with different types of tokenization. Following question: How is it managed for a sentence like “The capital of France is [MASK]” to predict Paris, assuming Paris would be split into two tokens by the sub-word-tokenizer (e.g. for BERT)? Because the model is only trained to predict single tokens, so is there any beam-search mechanic or similar to deal with multiple tokens per [MASK] or am I missing something else?

I already tried to make sense from the huggingface implementation source code but I did not really found the lines where this happens.

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![harnisph](https://avatars.discourse-cdn.com/v4/letter/h/f0a364/32.png) [@harnisph](https://discuss.huggingface.co/u/harnisph)\
**Post date:** [April 6, 2022, 4:18pm UTC](https://discuss.huggingface.co/t/how-does-fillmaskpipeline-work-with-subword-tokenization/15820/2 "2022-04-06T16:18:42Z")

</div>

> [@harnisph](#):
>
> nderstanding problem in regard to the FillMaskPipeline with different types of tokenization. Following question: How is it managed for a sentence like “The capital of France is [MASK]” to predict Paris, assuming Paris would be split into two tokens by the sub-word-tokenizer (e.g. for BERT)? Because the model is only trained to

After reading more of the literature and comparing the FillMaskPipeline against my own trivial implementation, I found out that the common approach only allows for single tokens and this is a known limitation. Although there are possible strategies to enable multi-token-prediction, they introduce a new layer of complexity and design choices.