# Customizing generation config for Trainer's training loop evaluation

**URL:** <https://discuss.huggingface.co/t/customizing-generation-config-for-trainers-training-loop-evaluation/71212>\
**Category:** 🤗Transformers\
**Created:** [January 30, 2024, 4:13pm UTC](https://discuss.huggingface.co/t/customizing-generation-config-for-trainers-training-loop-evaluation/71212 "2024-01-30T16:13:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kripi](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@Kripi](https://discuss.huggingface.co/u/Kripi)\
**Post date:** [January 30, 2024, 4:13pm UTC](https://discuss.huggingface.co/t/customizing-generation-config-for-trainers-training-loop-evaluation/71212/1 "2024-01-30T16:13:24Z")

</div>

When using Trainer’s training loop, I’d like to be able to customize the parameters for generation (used for evaluation). I tried the following approach:

```auto
gen_config = copy.deepcopy(model.generation_config)
gen_config.update(
    max_new_tokens=10,
    early_stopping=False,
    num_beams=1,
    no_repeat_ngram_size=0,
)
gen_config.validate()
training_args = Seq2SeqTrainingArguments(
    ....,
    predict_with_generate=True,
    generation_config=gen_config,
)
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    eval_dataset=...,
    ....
)
trainer.train()

```

However, the values are not used (for example, `max_new_tokens` is not respected). This warning is generated:

> Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file ([Text generation strategies](https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model)) instead. This warning will be raised to an exception in v4.41.  
> Non-default generation parameters: {‘early\_stopping’: True, ‘num\_beams’: 4, ‘no\_repeat\_ngram\_size’: 3, ‘forced\_bos\_token\_id’: 0, ‘forced\_eos\_token\_id’: 2}

along with:

> /opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py:1128: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.

I tried to follow the advice from the first warning by saving the modified configuration into a file and then loading it:

```auto
gen_config = copy.deepcopy(model.generation_config)
gen_config.update(
    max_new_tokens=10,
    early_stopping=False,
    num_beams=1,
    no_repeat_ngram_size=0,
)
gen_config_file = "gen_config.json"
gen_config.to_json_file(gen_config_file)
training_args = Seq2SeqTrainingArguments(
    ....,
    predict_with_generate=True,
    generation_config=gen_config_file,
)
...

```

but the warning stays the same.

How can I customize the evaluation generation when using Trainer? Thank you for any advice.

---

<div class="post-metadata">

**Author:** ![Kripi](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@Kripi](https://discuss.huggingface.co/u/Kripi)\
**Post date:** [January 31, 2024, 9:44am UTC](https://discuss.huggingface.co/t/customizing-generation-config-for-trainers-training-loop-evaluation/71212/2 "2024-01-31T09:44:45Z")

</div>

I found the problem. The warning was actually triggered by Trainer saving the model via `save_pretrained`, not by generation. I submitted an issue: [BART-base save\_pretrained triggers a warning about GenerationConfig · Issue #28793 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/28793)