# Trainer.evaluate() with text generation **URL:** \ **Category:** Beginners\ **Created:** [August 4, 2020, 4:51pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-with-text-generation/591 "2020-08-04T16:51:45Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![kstats](https://avatars.discourse-cdn.com/v4/letter/k/c77e96/32.png) [@kstats](https://discuss.huggingface.co/u/kstats)\ **Post date:** [August 4, 2020, 4:51pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-with-text-generation/591/1 "2020-08-04T16:51:45Z")
Hi everyone, I’m fine-tuning XLNet for generation. For training, I’ve edited the permutation\_mask to predict the target sequence one word at a time. I’m evaluating my trained model and am trying to decide between trainer.evaluate() and model.generate(). Running the same input/model with both methods yields different predicted tokens. Is it correct that trainer.evaluate() is not set up for sequential generation? I’ll switch my evaluation code to use model.generate() if that’s the case. Thanks for the help! ---
**Author:** ![rosafish](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rosafish](https://discuss.huggingface.co/u/rosafish)\ **Post date:** [August 11, 2020, 2:25pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-with-text-generation/591/2 "2020-08-11T14:25:47Z")
Hi, I encountered a similar problem when trying to use EncoderDecoderModel for seq2seq tasks. It seems like Trainer does not support text-generation tasks for now, as their website [https://huggingface.co/transformers/examples.html](https://huggingface.co/transformers/examples.html) shows. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 11, 2020, 2:28pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-with-text-generation/591/3 "2020-08-11T14:28:56Z")
There’s a [PR](https://github.com/huggingface/transformers/pull/5840) for that, you can try to use it. ---
**Author:** ![cgawron](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cgawron/32/28315_2.png) [@cgawron](https://discuss.huggingface.co/u/cgawron)\ **Post date:** [December 31, 2021, 9:42am UTC](https://discuss.huggingface.co/t/trainer-evaluate-with-text-generation/591/4 "2021-12-31T09:42:36Z")
Is there any update regarding this topic? I would like to train a `VisionEncoderDecoderModel` for image captioning and measure the BLEU metrics during evaluation. The `EvalPrediction` object I get in `compute_metrics` just contains the logits, not the generated texts or tokens (i.e. the result of a beam search). I would assume that the computation of metrics on the result of `generate` is not uncommon. The [PR](https://github.com/huggingface/transformers/pull/5840) mentioned in this thread seems to be stale and there have been quite some changes to `Trainer` since it was proposed. ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [December 31, 2021, 2:44pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-with-text-generation/591/5 "2021-12-31T14:44:17Z")
Hi @cgawron, you can take a look at my TrOCR notebooks here: [Transformers-Tutorials/TrOCR at master · NielsRogge/Transformers-Tutorials · GitHub](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/TrOCR). They include several example notebooks regarding fine-tuning TrOCR (which is an instance of `VisionEncoderDecoderModel`). I have a notebook regarding using the Seq2SeqTrainer, but also using native PyTorch. In both cases, I illustrate how to compute metrics using `generate` (in the notebooks I use CER, but you can easily replace it with something like ROUGE or BLEU). ---
**Author:** ![cgawron](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cgawron/32/28315_2.png) [@cgawron](https://discuss.huggingface.co/u/cgawron)\ **Post date:** [December 31, 2021, 7:57pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-with-text-generation/591/6 "2021-12-31T19:57:14Z")
Thank you, @nielsr, for providing these examples! Just for reference for other readers: The [`Seq2SeqTrainingArguments`](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Seq2SeqTrainingArguments) now contain a flag `predict_with_generate` exactly for this purpose.