# BertForMaskedLM train

**URL:** <https://discuss.huggingface.co/t/bertformaskedlm-train/2686>\
**Category:** 🤗Transformers\
**Created:** [December 8, 2020, 9:10am UTC](https://discuss.huggingface.co/t/bertformaskedlm-train/2686 "2020-12-08T09:10:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![yun](https://avatars.discourse-cdn.com/v4/letter/y/bcef8e/32.png) [@yun](https://discuss.huggingface.co/u/yun)\
**Post date:** [December 8, 2020, 9:10am UTC](https://discuss.huggingface.co/t/bertformaskedlm-train/2686/1 "2020-12-08T09:10:51Z")

</div>

I have a question  
When training using BertForMaskedLM, is the train data as below correct?

- token2idx

```auto
<pad> : 0, <mask>: 1, <cls>:2, <sep>:3

```

- max len : 8
- input token

```auto
 <cls> hello i <mask> cats <sep>

```

- input ids

```auto
 [2, 34,45,1,56,3,0,0]

```

- attention\_mask

```auto
 [1,1,1,1,1,1,0,0]

```

- labels

```auto
 [-100,-100,-100,64,-100,-100,-100,-100]

```

I wonder if I should also assign -100 to labels for padding token.

---

<div class="post-metadata">

**Author:** ![ayalaall](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@ayalaall](https://discuss.huggingface.co/u/ayalaall)\
**Post date:** [January 19, 2021, 10:38am UTC](https://discuss.huggingface.co/t/bertformaskedlm-train/2686/2 "2021-01-19T10:38:05Z")

</div>

Hi,  
Were you able to figure it out? I’m also trying to do the same thing.

Thanks,  
Ayala

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [January 20, 2021, 8:05am UTC](https://discuss.huggingface.co/t/bertformaskedlm-train/2686/3 "2021-01-20T08:05:18Z")

</div>

you should replace all tokens (including paddding) in `labels` with -100 except the masked tokens so the loss will only be calculated for masked tokens.