# Masked language model for BART (Not BERT)

**URL:** <https://discuss.huggingface.co/t/masked-language-model-for-bart-not-bert/19945>\
**Category:** 🤗Transformers\
**Created:** [July 3, 2022, 11:33am UTC](https://discuss.huggingface.co/t/masked-language-model-for-bart-not-bert/19945 "2022-07-03T11:33:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahadda5](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@ahadda5](https://discuss.huggingface.co/u/ahadda5)\
**Post date:** [July 3, 2022, 11:33am UTC](https://discuss.huggingface.co/t/masked-language-model-for-bart-not-bert/19945/1 "2022-07-03T11:33:07Z")

</div>

Hi, I’m trying to train a BART model using masking(MLM).  
The model type is BartForConditionalGeneration. The task I have is text generation(key phrases) of an input text.

Before trying it on a custom dataset, I wanted to try it on the given official huggingface example [here](https://huggingface.co/course/chapter7/3?fw=pt#the-dataset), which is in fact similar to [huggingface github example](https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)

To save space and not past the entire code as is, I changed the model to one suited for my task i found on huggingface. [**Everything else is the same,** plus enabling this variable to help in CUDA stack debuging _os.environ[‘CUDA\_LAUNCH\_BLOCKING’]=“1”_ ]

`model_checkpoint = "distilbert-base-uncased" model = AutoModelForMaskedLM.from_pretrained(model_checkpoint) `  
to

```auto
model_checkpoint = "memray/bart-wikikp"
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)

```

Based on provided documentation, this unsupervised approach is viable if one wants to fine-tune the model for a specific domain. Therefore, before fine-tuning, Masked language modelling helps acquaint the model with the new corpus first. Also, in the documentation there is no mention of specific-tasks, e.g. only applicable for QA(question-answering) or text classification.

Please note that in the first link above the imdb dataset is used.

The errors i get are CUDA related, (using GPU) when training

```auto
trainer.train()

```

**Error**  
“\*\*\*\*\* Running training \*\*\*\*\*  
Num examples = 10000  
Num Epochs = 3  
Instantaneous batch size per device = 32  
Total train batch size (w. parallel, distributed & accumulation) = 32  
Gradient Accumulation steps = 1  
Total optimization steps = 939  
0%| | 0/939 [00:00\<?, ?it/s]/opt/conda/conda-bld/pytorch\_1646755953518/work/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [42,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.  
/opt/conda/conda-bld/pytorch\_1646755953518/work/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [42,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.  
/opt/conda/conda-bld/pytorch\_1646755953518/work/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [42,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

Traceback (most recent call last):  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/IPython/core/interactiveshell.py”, line 3457, in run\_code  
exec(code\_obj, self.user\_global\_ns, self.user\_ns)  
File “”, line 1, in   
trainer.train()  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/trainer.py”, line 1413, in train  
ignore\_keys\_for\_eval=ignore\_keys\_for\_eval,  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/trainer.py”, line 1651, in \_inner\_training\_loop  
tr\_loss\_step = self.training\_step(model, inputs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/trainer.py”, line 2345, in training\_step  
loss = self.compute\_loss(model, inputs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/trainer.py”, line 2377, in compute\_loss  
outputs = model(\*\*inputs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in \_call\_impl  
return forward\_call(\*input, \*\*kwargs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/models/bart/modeling\_bart.py”, line 1368, in forward  
return\_dict=return\_dict,  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in \_call\_impl  
return forward\_call(\*input, \*\*kwargs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/models/bart/modeling\_bart.py”, line 1229, in forward  
return\_dict=return\_dict,  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in \_call\_impl  
return forward\_call(\*input, \*\*kwargs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/models/bart/modeling\_bart.py”, line 850, in forward  
output\_attentions=output\_attentions,  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in \_call\_impl  
return forward\_call(\*input, \*\*kwargs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/models/bart/modeling\_bart.py”, line 327, in forward  
output\_attentions=output\_attentions,  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in \_call\_impl  
return forward\_call(\*input, \*\*kwargs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/transformers/models/bart/modeling\_bart.py”, line 191, in forward  
query\_states = self.q\_proj(hidden\_states) \* self.scaling  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in \_call\_impl  
return forward\_call(\*input, \*\*kwargs)  
File “/home/haddad/.conda/envs/hugg/lib/python3.7/site-packages/torch/nn/modules/linear.py”, line 103, in forward  
return F.linear(input, self.weight, self.bias)  
RuntimeError: CUDA error: CUBLAS\_STATUS\_NOT\_INITIALIZED when calling `cublasCreate(handle)`  
/”

---

<div class="post-metadata">

**Author:** ![ahadda5](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@ahadda5](https://discuss.huggingface.co/u/ahadda5)\
**Post date:** [July 3, 2022, 11:35am UTC](https://discuss.huggingface.co/t/masked-language-model-for-bart-not-bert/19945/2 "2022-07-03T11:35:46Z")

</div>

If using accelerator module, i get

```auto
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.)

```

Everything I read says that there is a mismatch between the model head, input tensor or other tensors causing this to happen. It seems BART needs something extra to be adapted to MLM?!

**[UPDATE]**  
So commenting the collator argument gets the trainer.train() to work, otherwise CUDA faces lots of issues whether with the accelerator or not.

```auto
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=downsampled_dataset["train"],
    eval_dataset=downsampled_dataset["test"],
    # data_collator=data_collator,
)

```

---

<div class="post-metadata">

**Author:** ![ahadda5](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@ahadda5](https://discuss.huggingface.co/u/ahadda5)\
**Post date:** [July 5, 2022, 7:44am UTC](https://discuss.huggingface.co/t/masked-language-model-for-bart-not-bert/19945/3 "2022-07-05T07:44:18Z")

</div>

One thing i suspect, is that the model input embeddings is the vocab size (50265) x 1024!  
Whereas the imdb is chunked into 128.  
How can i change the model to adopt that input dimension? 1024 → 128

---

<div class="post-metadata">

**Author:** ![cog](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@cog](https://discuss.huggingface.co/u/cog)\
**Post date:** [July 5, 2022, 10:21am UTC](https://discuss.huggingface.co/t/masked-language-model-for-bart-not-bert/19945/4 "2022-07-05T10:21:08Z")

</div>

hi @ahadda5 ,

is there config setting different?

setting config’s max length size or hidden layer dimension.

[HF bart config docs](https://huggingface.co/docs/transformers/model_doc/bart#transformers.BartConfig)

Also, if you want to build BART for Masked LM, add some last layers to predict hidden layer’s output to your output dimension.

for example, in [BertForMaskedLM](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForMaskedLM), class BertLMPredictionHead(nn.Module) fit dimension of hidden to output(vocab)

`self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)`

you can get hint from [BertForMaskedLM](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForMaskedLM) 's layer structure to build BART Masked LM .

hope to help.

---

<div class="post-metadata">

**Author:** ![ahadda5](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@ahadda5](https://discuss.huggingface.co/u/ahadda5)\
**Post date:** [July 5, 2022, 1:24pm UTC](https://discuss.huggingface.co/t/masked-language-model-for-bart-not-bert/19945/5 "2022-07-05T13:24:14Z")

</div>

thanks for your reply. I will investigate further.  
But debugging the BART beast,  
encoder layer dims are consistent with decoder layers.  
Labels are also consistent with input ids for Masking.

I’ll take a look at BERT

---

<div class="post-metadata">

**Author:** ![ahadda5](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@ahadda5](https://discuss.huggingface.co/u/ahadda5)\
**Post date:** [July 5, 2022, 5:18pm UTC](https://discuss.huggingface.co/t/masked-language-model-for-bart-not-bert/19945/6 "2022-07-05T17:18:29Z")

</div>

Okay so the issue was that the used model has a vocab size, 50264! however the tokenizer has a size of 50265!

So had to resize\_token\_embedding the model to that of tokenizer!  
thanks @sgugger , @cog for the guidance. Happy coding 🙂 !