# Bert output for padding tokens

**URL:** <https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550>\
**Category:** Beginners\
**Created:** [October 15, 2020, 12:23pm UTC](https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550 "2020-10-15T12:23:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![datistiquo](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@datistiquo](https://discuss.huggingface.co/u/datistiquo)\
**Post date:** [October 15, 2020, 12:23pm UTC](https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550/1 "2020-10-15T12:23:34Z")

</div>

Hi,

I just saw that I have still embeddings of padding tokens in my sentence. I assumes that the BERT output would be a 768 dim 0 vector.

So if I feed sentences with max length of 20 into TFBert-Model I get for every of the 20 tokens an embedding different from 0 despite if the sentences is just of length 10 and the rest is padded.

How can I get output of BERT which ignores the padded output?

I thought that is why I feed the attention masks such paddings are ignored?

Maybe I misunderstand something?

**If I feed the sequence of hidden states of the output of BERT to a GlobalAveragePooling-Layer, do I need the masking tensor to avoid using padding embeddings?**

---

<div class="post-metadata">

**Author:** ![datistiquo](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@datistiquo](https://discuss.huggingface.co/u/datistiquo)\
**Post date:** [October 26, 2020, 3:08pm UTC](https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550/2 "2020-10-26T15:08:32Z")

</div>

Hey,

I think I should use the attention\_masking from the tokenizer for use in GlobalAveragePooling1d(x, mask=mask) , right?

Otherwise, the average of all token (padded tokens included) is built, right?

---

<div class="post-metadata">

**Author:** ![datistiquo](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@datistiquo](https://discuss.huggingface.co/u/datistiquo)\
**Post date:** [October 27, 2020, 3:16pm UTC](https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550/3 "2020-10-27T15:16:55Z")

</div>

As you use GlobalAveragePooling inside your model and train it, normally in tf.keras masking is propagated.

So, can I assume that masking is propagated from the BERT model towards this pooling layer?

Any ideas? And how could I check this?

---

<div class="post-metadata">

**Author:** ![balvisio](https://avatars.discourse-cdn.com/v4/letter/b/b19c9b/32.png) [@balvisio](https://discuss.huggingface.co/u/balvisio)\
**Post date:** [February 22, 2023, 3:11am UTC](https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550/4 "2023-02-22T03:11:43Z")

</div>

@datistiquo : You are correct in that you should use the “attention\_mask” from the tokenizer in:

`GlobalAveragePooling1d(x, mask=mask)`.

To answer your other question: No, the mask won’t be propagated from the output of BERT (“TFBertMainLayer”) since by looking at the source code of [TFBertMainLayer](https://github.com/huggingface/transformers/blob/bb5a2f2fc30985841289207b9f1f7765d8abc4e0/src/transformers/models/bert/modeling_tf_bert.py#L720), it doesn’t support masking. According to the keras [documentation](https://www.tensorflow.org/guide/keras/masking_and_padding#opting-in_to_mask_propagation_on_compatible_layers) here, it should set “self.supports\_mask=True” in the “ **init** ”" method.

Thus you need to set the mask explicitly. Here is an example:

```python
from keras.layers import GlobalAveragePooling1D
from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained("bert-base-uncased")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

x = GlobalAveragePooling1D()(output[0], encoded_input["attention_mask"]) 

```

Hope it helps