# Is attention\_mask implemented correctly in BERT?

**URL:** <https://discuss.huggingface.co/t/is-attention-mask-implemented-correctly-in-bert/34138>\
**Category:** 🤗Transformers\
**Created:** [March 19, 2023, 3:00am UTC](https://discuss.huggingface.co/t/is-attention-mask-implemented-correctly-in-bert/34138 "2023-03-19T03:00:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bjlkeng](https://avatars.discourse-cdn.com/v4/letter/b/ea5d25/32.png) [@bjlkeng](https://discuss.huggingface.co/u/bjlkeng)\
**Post date:** [March 19, 2023, 3:00am UTC](https://discuss.huggingface.co/t/is-attention-mask-implemented-correctly-in-bert/34138/1 "2023-03-19T03:00:23Z")

</div>

I was browsing through the Bert model code and noticed that the attention\_mask is implemented as a simple addition:

> <https://github.com/huggingface/transformers/blob/v4.27.1/src/transformers/models/bert/modeling_bert.py#L350>

This looks strange to me because in the original implementation they map 0’s to -10000 (pre-softmax):

> <https://github.com/google-research/bert/blob/master/modeling.py#L712>

I searched through the file but couldn’t find an equivalent mapping so it kind of looks like it’s just adding the attention mask to the logits. Am I missing something?

---

<div class="post-metadata">

**Author:** ![kakahw](https://avatars.discourse-cdn.com/v4/letter/k/aeb1de/32.png) [@kakahw](https://discuss.huggingface.co/u/kakahw)\
**Post date:** [November 12, 2023, 12:55am UTC](https://discuss.huggingface.co/t/is-attention-mask-implemented-correctly-in-bert/34138/2 "2023-11-12T00:55:30Z")

</div>

In BertModel.forward, it calls into ModuleUtilsMixin.get\_extended\_attention\_mask to update the attention mask which inverted the values in this line:

```auto
extended_attention_mask = (1.0 - extended_attention_mask) * torch.finfo(dtype).min

```

---

<div class="post-metadata">

**Author:** ![bjlkeng](https://avatars.discourse-cdn.com/v4/letter/b/ea5d25/32.png) [@bjlkeng](https://discuss.huggingface.co/u/bjlkeng)\
**Post date:** [November 12, 2023, 4:05pm UTC](https://discuss.huggingface.co/t/is-attention-mask-implemented-correctly-in-bert/34138/3 "2023-11-12T16:05:15Z")

</div>

Thanks! I didn’t realize I was just looking at the sub-layer and not the entire model.

For those in the future, here are the two lines that are relevant:

> <https://github.com/huggingface/transformers/blob/v4.27.1/src/transformers/models/bert/modeling_bert.py#L993>

> <https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L935>