# Longformer's attention\_mask

**URL:** <https://discuss.huggingface.co/t/longformers-attention-mask/926>\
**Category:** Beginners\
**Created:** [August 30, 2020, 10:21am UTC](https://discuss.huggingface.co/t/longformers-attention-mask/926 "2020-08-30T10:21:10Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![fanluo](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@fanluo](https://discuss.huggingface.co/u/fanluo)\
**Post date:** [August 30, 2020, 10:21am UTC](https://discuss.huggingface.co/t/longformers-attention-mask/926/1 "2020-08-30T10:21:10Z")

</div>

Inside of forward function of [class LongformerSelfAttention(nn.Module)](https://github.com/allenai/longformer/blob/5a2b9da317c3407293b5069417a3556a5b15e28a/longformer/longformer.py#L89):

> The `attention_mask` is changed in `BertModel.forward` from 0, 1, 2 to  
> -ve: no attention  
> 0: local attention  
> +ve: global attention

I wonder how this change happens.

- Longformer inherits from RobertaModel, which inherits from [BertModel](https://huggingface.co/transformers/_modules/transformers/modeling_roberta.html#RobertaModel). Would BertModel’s forward also be called when calling Longformer 's forward function?
- How BertModel’s forward change 0, 1, 2 to -ve/0/+ve？

Thank you