# Positional encoding error in RoBERTa

**URL:** <https://discuss.huggingface.co/t/positional-encoding-error-in-roberta/57023>\
**Category:** 🤗Transformers\
**Created:** [October 1, 2023, 7:40pm UTC](https://discuss.huggingface.co/t/positional-encoding-error-in-roberta/57023 "2023-10-01T19:40:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pepper08](https://avatars.discourse-cdn.com/v4/letter/p/f9ae1b/32.png) [@Pepper08](https://discuss.huggingface.co/u/Pepper08)\
**Post date:** [October 1, 2023, 7:40pm UTC](https://discuss.huggingface.co/t/positional-encoding-error-in-roberta/57023/1 "2023-10-01T19:40:23Z")

</div>

Hello all,

I’m using a RobertaForMaskedLM model initialized with the following configuration:

```python
config = RobertaConfig(vocab_size = tokenizer.vocab_size, max_position_embeddings = 128)

```

because I padded my input tokens to a size of 128.

However, during the training, I get the following error message:

```python
IndexError: index out of range in self

```

which comes from the position\_ids in the position\_embedding layer.

Indeed, the positional embedding layer is initialized with the argument max\_position\_embeddings (equals to 128 here):

```python
self.position_embeddings = nn.Embedding(config.max_position_embeddings, ...)

```

whereas the function which creates position\_ids generates numbers from padding\_idx to padding\_idx + max\_position\_embeddings (so 129 \> 127).

Therefore, it seems that one needs to choose model\_max\_length = (128 + padding\_idx + 1) to avoid errors, but I can’t see any explanation of this in the documentation. Can anyone explain what I haven’t understood?

---

<div class="post-metadata">

**Author:** ![Sandy1857](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Sandy1857](https://discuss.huggingface.co/u/Sandy1857)\
**Post date:** [October 2, 2023, 12:52pm UTC](https://discuss.huggingface.co/t/positional-encoding-error-in-roberta/57023/2 "2023-10-02T12:52:30Z")

</div>

When you tokenize a text, a bos token and eos token is added to the beginning and end of the tokenized output.

So if your text has \>= 128 tokens, with truncation set to true, the tokens will be truncated to 126 so that there is space to add the bos and eos tokens. The extra 2 comes from this.