# Encoder-Decoder model only generates bos\_token's \[\<s\>\<s\>\<s\>\]

**URL:** <https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470>\
**Category:** Models\
**Created:** [November 21, 2022, 8:30pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470 "2022-11-21T20:30:09Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![AbdelrahmanZ](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)\
**Post date:** [November 21, 2022, 8:30pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/1 "2022-11-21T20:30:09Z")

</div>

Hello,

I fine-tuned bert2bert like model as encoder-decoder for translation task, the model used is `microsoft/Multilingual-MiniLM-L12-H384` with XLMRobertaTokenizer,  
When I run the fine-tuned model using generate() method, it only generates `bos_token` until it reaches limit of decoding length,  
The `decoder_start_token_id` is set the same as `bos_token_id`  
I’ve checked some related Github issues, but they were for brat model.

> <https://github.com/huggingface/transformers/issues/15559>
>
> \## Environment info
> \<!-- You can run the command \`transformers-cli env\` and cop…y-and-paste its output below.
> Don't forget to fill out the missing fields in that output! --\>
> 
> \- \`transformers\` version: 4.16.2 (issue exists on 4.9.2)
> \- Platform: Linux-4.4.0-210-generic-x86\_64-with-glibc2.10
> \- Python version: 3.8.10
> \- PyTorch version (GPU?): 1.8.1+cpu (False)
> \- Tensorflow version (GPU?): 2.3.1 (False)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> \- Using GPU in script?: \<fill in\>
> \- Using distributed or parallel set-up in script?: \<fill in\>
> 
> 
> \### Who can help
> @patrickvonplaten @sshleifer 
> 
> \## Information
> 
> Essentially re-opening \[issue 8005\](https://github.com/huggingface/transformers/issues/8005), BART-large does not mask fill properly (whereas BART-base has entirely reasonable outputs). The previous fix of setting \`force\_bos\_token\_to\_be\_generated = True\` is no longer viable since the option no longer exists in BART config. It also seems like adjust\_logits\_during\_generation (where force\_bos\_token\_to\_be\_generated was used) is no longer implemented in the BART model.
> 
> \## To reproduce
> 
> Steps to reproduce the behavior:
> \`\`\`
> tokenizer = BartTokenizer.from\_pretrained("facebook/bart-base", forced\_bos\_token\_id=0)
> model = BartForConditionalGeneration.from\_pretrained("facebook/bart-base")
> batch = tokenizer("My friends are \<mask\> but they eat too many carbs.", return\_tensors="pt")
> generated\_ids = model.generate(batch\["input\_ids"\])
> print(tokenizer.decode(generated\_ids\[0\]))
> \# Output: \</s\>\<s\>My friends are healthy, but they eat too many carbs.\</s\>
> 
> tokenizer = BartTokenizer.from\_pretrained("facebook/bart-large", forced\_bos\_token\_id=0)
> model = BartForConditionalGeneration.from\_pretrained("facebook/bart-large")
> batch = tokenizer("My friends are \<mask\> but they eat too many carbs.", return\_tensors="pt")
> generated\_ids = model.generate(batch\["input\_ids"\])
> print(tokenizer.decode(generated\_ids\[0\]))
> \# Output: \</s\>My,, but they eat too many carbs.\</s\>\`\`\`

> <https://github.com/huggingface/transformers/issues/18190>
>
> \### System Info
> 
> \- \`transformers\` version: 4.20.0.dev0
> \- Platform: Linux-4.18….0-348.23.1.el8\_5.x86\_64-x86\_64-with-centos-8.6-Green\_Obsidian
> \- Python version: 3.7.13
> \- Huggingface\_hub version: 0.7.0
> \- PyTorch version (GPU?): 1.11.0 (True)
> \- Tensorflow version (GPU?): not installed (NA)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> 
> 
> \### Who can help?
> 
> @ydshieh 
> 
> \### Information
> 
> \- \[X\] The official example scripts
> \- \[\] My own modified scripts
> 
> \### Tasks
> 
> \- \[X\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> \`\`\`
> OUTPUT\_DIR=/home/ratish/project
> python -m torch.distributed.launch --nproc\_per\_node=1 examples/pytorch/summarization/run\_summarization.py \\
> --model\_name\_or\_path allenai/led-large-16384 \\
> --do\_train \\
> --do\_eval \\
> --dataset\_name xsum \\
> --output\_dir ${OUTPUT\_DIR} \\
> --per\_device\_train\_batch\_size=4 \\
> --per\_device\_eval\_batch\_size=4 \\
> --overwrite\_output\_dir \\
> --predict\_with\_generate \\
> --overwrite\_output\_dir \\
> --logging\_dir logs \\
> --evaluation\_strategy steps \\
> --eval\_steps 100 \\
> --logging\_steps 100 \\
> --report\_to tensorboard \\
> --save\_total\_limit 5 \\
> --save\_steps 100 \\
> --load\_best\_model\_at\_end \\
> --greater\_is\_better True \\
> --metric\_for\_best\_model rougeL \\
> --max\_eval\_samples 100 \\
> --num\_beams 3
> \`\`\`
> The logs shows that at checkpoint 1800 the rouge becomes zero. 
> \`{'eval\_loss': 2.172360897064209, 'eval\_rouge1': 0.0, 'eval\_rouge2': 0.0, 'eval\_rougeL': 0.0, 'eval\_rougeLsum': 0.0, 'eval\_gen\_len': 20.0, 'eval\_runtime': 10.2823, 'eval\_samples\_per\_second': 9.725, 'eval\_steps\_per\_second': 2.431, 'epoch': 0.04}\`
> 
> I evaluate the model output using the below function:
> \`\`\`
> def generate\_output():
> import torch
> from transformers import LEDTokenizer, LEDForConditionalGeneration
> MODEL="/home/ratish/checkpoint-1800"
> model = LEDForConditionalGeneration.from\_pretrained(MODEL)
> tokenizer = LEDTokenizer.from\_pretrained(MODEL)
> ARTICLE\_TO\_SUMMARIZE = "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
> inputs = tokenizer.encode(ARTICLE\_TO\_SUMMARIZE, return\_tensors="pt")
> global\_attention\_mask = torch.zeros\_like(inputs)
> global\_attention\_mask\[:, 0\] = 1
> summary\_ids = model.generate(inputs, global\_attention\_mask=global\_attention\_mask, num\_beams=3, max\_length=32)
> print(tokenizer.decode(summary\_ids\[0\], skip\_special\_tokens=False, clean\_up\_tokenization\_spaces=False))
> \`\`\`
> It produces the output \`\</s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\<s\>\`
> 
> \### Expected behavior
> 
> The model should produce the summary of the news article.

and the solutions proposed didn’t work for my model,

Any suggestions or ideas would be appreciated,  
Thanks.

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [November 23, 2022, 8:01am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/2 "2022-11-23T08:01:02Z")

</div>

Hi @AbdelrahmanZ , could you provide a script (or the command you used to launch the training) that could reproduce the issue, please?

---

<div class="post-metadata">

**Author:** ![AbdelrahmanZ](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)\
**Post date:** [November 23, 2022, 8:55am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/3 "2022-11-23T08:55:57Z")

</div>

Actually, the training script is a little bit huge, though I will share the most important parts:  
Loading the model:

```auto
            tokenizer = XLMRobertaTokenizer.from_pretrained("microsoft/Multilingual-MiniLM-L12-H384")
            tokenizer.bos_token = tokenizer.cls_token
            tokenizer.eos_token = tokenizer.sep_token
            encoder_decoder_model = EncoderDecoderModel.from_encoder_decoder_pretrained(
                "microsoft/Multilingual-MiniLM-L12-H384",
                "microsoft/Multilingual-MiniLM-L12-H384",
            )
            encoder_decoder_model.config.decoder_start_token_id = tokenizer.bos_token_id
            encoder_decoder_model.config.eos_token_id = tokenizer.eos_token_id
            encoder_decoder_model.config.pad_token_id = tokenizer.pad_token_id

```

Single (source, target) example tokenization:

```auto
        model_inputs = self.tokenizer(
            examples["source"].strip(),
            max_length=self.params["encoder_max_length"],
            padding=False,
            truncation=True,
        )
        targets = self.tokenizer(
            examples["target"].strip(),
            max_length=self.params["decoder_max_length"],
            padding=False,
            truncation=True,
        )
        model_inputs["labels"] = targets["input_ids"]

```

Then I create Seq2SeqTrainer, and train the model.  
For inference, this is the generation config:

```auto
generation_config = dict(
    max_length=None,
    min_length=None,
    do_sample=False,
    early_stopping=True,
    num_beams=1,
    temperature=1.0,
    top_k=None,
    top_p=None,
    length_penalty=1.0, # > 1.0 longer sequences, < 1.0 shorter sequences
    num_return_sequences=1,
    max_time=None, # in seconds
    num_beam_groups=1,
    output_scores=False,
)

```

call to generate():

```auto
    inputs = tokenizer(text, return_tensors="pt")
    input_ids = inputs.input_ids
    generated_texts = model.generate(input_ids, **generation_config)

```

I hope this is enough to reproduce the issue. Thank you @ydshieh .

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [November 24, 2022, 4:13pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/4 "2022-11-24T16:13:20Z")

</div>

Hi, @AbdelrahmanZ Thank you for the effort. It would still be much easier if a single training script could be provided. Do you mind to copy the code and paste in a Google Colab notebook (in a single cell), or even put your script in a GitHub repository. Otherwise, you can send me an email if no other ways fit. Thank you!

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [November 28, 2022, 1:46pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/5 "2022-11-28T13:46:54Z")

</div>

Continue the discussion here.

From @ [AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)

> I just shared the notebook, I trained the model on Arabic to English translation task. generate() works fine, and the issue doesn’t appear again.  
> The issue of generating `[<s> <s> <s>...]` happened with me for training on Arabic to Arabic task. I think it could be related to:

> - The type of source and target languages
> - Maybe it appears only after a large number of training steps
> - I should rerun the finetuning and check the checkpoints gradually, unfortunately this would be painful to me as I need to wait at least two weeks to finish.

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [November 28, 2022, 1:53pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/6 "2022-11-28T13:53:07Z")

</div>

Hi @AbdelrahmanZ

In the notebook, you have `num_train_epochs=2`, but you mentioned `Maybe it appears only after a large number of training steps`? What’s the `num_train_epochs` you used when you train on Arabic to Arabic (for which you get the issue)?

Would you be able to reproduce with `Arabic to Arabic` but keeping everything else the same (with the notebook)?

From your description, it seems there is no real bug in our codebase, but it is still interesting to try to figure out why you got the issue from the 1st place - if you can find the situation that can reproduce it, and see if there is indeed something we can/should fix.

Thank you!

---

<div class="post-metadata">

**Author:** ![AbdelrahmanZ](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)\
**Post date:** [November 28, 2022, 2:01pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/7 "2022-11-28T14:01:49Z")

</div>

Hi @ydshieh

Regarding the `num_train_epochs` it was just an assumption, I’m not sure if this is the cause.  
I will try to rerun the finetuning and modify some hyper-parameters, hopefully the issue will be reproduced.

---

<div class="post-metadata">

**Author:** ![AbdelrahmanZ](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)\
**Post date:** [November 29, 2022, 10:35am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/8 "2022-11-29T10:35:20Z")

</div>

Hi @ydshieh

I think the issue is reproduced. The issue appeared when the model got initialized from a checkpoint.  
Note this didn’t happen on the same model instance that got fine-tuned, only after initializing the model from a checkpoint.  
So I believe the steps to reproduce the issue are:

1. Run the fine-tuning script(the notebook previously shared).
2. Save a checkpoint.
3. Run the following script that initialize a new instance of the model from the saved checkpoint.

```auto
import torch
from transformers import (
    EncoderDecoderModel,
    XLMRobertaTokenizer,
)
generation_config = dict(
    max_length=None,
    min_length=None,
    do_sample=False,
    early_stopping=True,
    num_beams=1,
    temperature=1.0,
    top_k=None,
    top_p=None,
    length_penalty=1.0, # > 1.0 longer sequences, < 1.0 shorter sequences
    num_return_sequences=1,
    max_time=None, # in seconds
    num_beam_groups=1,
    output_scores=False,
)

model_path = "checkpoint-path"
tokenizer = XLMRobertaTokenizer.from_pretrained(model_path)
model = EncoderDecoderModel.from_pretrained(model_path)
model.config.vocab_size = model.config.decoder.vocab_size
model = model.eval()
text = "أجرت فرقة العمل، يشترك في رئاسته جيف موس وآلان Paller، مقابلات مكثفة مع خبراء من الحكومة والقطاع الخاص، والأوساط الأكاديمية في تطوير توصياتها لتنمية المهارات الفنية المتقدمة للأمن السيبراني القوى العاملة DHS وتوسيع خط أنابيب الوطني للرجال والنساء مع هذه المهارات الأمن السيبراني."
inputs = tokenizer(text, return_tensors="pt")
input_ids = inputs.input_ids
generated_texts = model.generate(input_ids, **generation_config)
print(generated_texts)

```

requirements:

```auto
transformers==4.17.0
torch==1.10.1
sentencepiece==0.1.96

```

I also tried latest transformers version, but still the same behavior.

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [November 29, 2022, 11:57am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/9 "2022-11-29T11:57:15Z")

</div>

Hi Abdelrahman Alzboon!

Thank you a lot for the effort. Could you post your finding above to the forum thread that you created.  
We will definitely look into this issue.

Cheers,  
Yih-Dar

Abdelrahman Alzboon via Hugging Face Forums \<[notifications@hellohellohello.discoursemail.com](mailto:notifications@hellohellohello.discoursemail.com)\> 於 2022年11月29日 週二 上午11:45寫道：

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [November 30, 2022, 3:48pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/10 "2022-11-30T15:48:21Z")

</div>

I can reproduce the issue with the notebook - with smaller datasets/short sequences. In my case, no need to save and reload, the issue occurs after some training steps.

Will definitely look into this.

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [December 1, 2022, 3:44pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/11 "2022-12-01T15:44:53Z")

</div>

Hi , @AbdelrahmanZ

Have you tried to

```auto
  model_inputs["labels"] = model_inputs["labels"][1:]

```

after

```auto
  model_inputs["labels"] = targets["input_ids"]

```

inside

```auto
preprocess_function

```

I modified your script a bit for faster debugging, and adding this extra line works for me.

I would suggest you to try the fix with a smaller subset of the dataset, short sequence, fewer steps to make sure it works on your side before launching large training.

Here is the modified notebook

> **[Google Colaboratory](https://colab.research.google.com/drive/1yrVcYGyRAd4AZfv9HZ-wmXv31bE586Lj?usp=sharing)**

---

<div class="post-metadata">

**Author:** ![AbdelrahmanZ](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)\
**Post date:** [December 2, 2022, 9:21pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/12 "2022-12-02T21:21:06Z")

</div>

Hi @ydshieh ,

Great catch, unfortunately it requires refine-tuning the model from scratch.  
I’ve used the same script before with other bert-like models and it works well.  
Is the problem related to fine-tuning and input-preparation?  
Can we change some stuff in the generation so it would generate text normally?

Thank you.

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [December 3, 2022, 8:49am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/13 "2022-12-03T08:49:15Z")

</div>

Hi. I think the training (fine-tuning) is broken unfortunately. If you look a few examples from `model_inputs["labels"]` in `preprocess_function`, your original script will have leading `<bos>` (token id `0`). And the decoder inputs will be prepared as `<decoder_start (0)> <bos (0)> ...` , and therefore `<decoder_start (0)>` is trained to predict `<bos (0)>`. That’s the main cause of the issue.

I don’t know why it works sometimes however. BTW, would you mind sharing what kind of bert models you trained which work?

Thank you and good luck!

---

<div class="post-metadata">

**Author:** ![AbdelrahmanZ](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)\
**Post date:** [December 4, 2022, 10:42am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/14 "2022-12-04T10:42:07Z")

</div>

Hi @ydshieh ,

So the model will learn to generate `<bos (0)>` if given input is `0` and so on till the max length.

Though as I told you before, the same preprocessing was also applied to other models, for example `lanwuwei/GigaBERT-v4-Arabic-and-English`, for this model the `decoder_start_token_id=2` and tokens generation for the first steps are the same between the two models, I mean `GigaBERT` also predicts `<bos (2)>` for `<decoder_start (2)>` but after this step it starts generating tokens as expected.

I’ve debugged the `generate()` method and compare the generation steps of the two models, everything the same, but when it comes to generating next token; MiniLM model generates bos token’s instead.

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [December 4, 2022, 11:11am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/15 "2022-12-04T11:11:08Z")

</div>

Hi @AbdelrahmanZ

I see. Notice that even it works for some models, we already see a few times this (similar) issue occurs previously (for Bart/LongFormer, etc, although the cause is somehow different).

But in your case here, you `decoder_start_token_id` is the same as `bos_token_id`, but also the same as the `pad_token_id`. When `generate` method generates a `pad_token_id`, there is some logic to say the generation reaches the end, and it won’t generate any new token but padding. This is probably the reason why it doesn’t work in your specific case for MiniLM.

---

<div class="post-metadata">

**Author:** ![AbdelrahmanZ](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)\
**Post date:** [December 4, 2022, 11:21am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/16 "2022-12-04T11:21:09Z")

</div>

Alright, it seems I need to refine-tune the model again. I will mark the issue as solved.

Thank you @ydshieh, I really appreciate your help, if you need any further information or need to debug the issue further I’ll be glad to help. Good Luck.

---

<div class="post-metadata">

**Author:** ![ydshieh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ydshieh/32/17513_2.png) [@ydshieh](https://discuss.huggingface.co/u/ydshieh)\
**Post date:** [December 4, 2022, 11:30am UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/17 "2022-12-04T11:30:42Z")

</div>

Just want to share (or reiterate) a tip: It’s always a good idea to run the training with a much smaller subset of the dataset, with much fewer training steps, as the one I shared previoiusly. (I know, you have used the same training script for other models 🙂 where it works!)

It’s also a good idea to modify the training script to perform some generateion when saving the checkpoints - and save the generation results for investigation - so we can spot any issue earlier.

Good luck!

---

<div class="post-metadata">

**Author:** ![AbdelrahmanZ](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@AbdelrahmanZ](https://discuss.huggingface.co/u/AbdelrahmanZ)\
**Post date:** [December 6, 2022, 1:15pm UTC](https://discuss.huggingface.co/t/encoder-decoder-model-only-generates-bos-tokens-s-s-s/26470/18 "2022-12-06T13:15:18Z")

</div>

I just found a parameter for generate(): **[begin\_suppress\_tokens](https://huggingface.co/docs/transformers/v4.25.1/en/main_classes/text_generation#transformers.GenerationMixin.generate.begin_suppress_tokens)**(transformers==v4.25.1)  
that prevents the generation of `bos token` at the start, I tried it and now the model generates sequences normally, though I’m not sure if it will affect the overall performance.