# Seq2seq evaluation speed is slow **URL:** \ **Category:** šŸ¤—Transformers\ **Created:** [July 14, 2020, 5:12pm UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282 "2020-07-14T17:12:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 14, 2020, 5:12pm UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282/1 "2020-07-14T17:12:16Z")
While running the seq2seq examples following the Readme I found that training is relatively fast and uses \>50% of the GPU while evaluations (with the exact same batch size) is painfully slow with low GPU utilization. True for both T5 and BART. What am I doing wrong? ---
**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\ **Post date:** [July 15, 2020, 11:57am UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282/2 "2020-07-15T11:57:18Z")
As a guess, evaluation requires a generation procedure whereas training uses teacher-forcing and cross-entropy loss. This might not be the case though because the validation step may include some generation - I would highly recommend using the Pycharm debugger to test out what part of the code is taking so long. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 15, 2020, 12:06pm UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282/3 "2020-07-15T12:06:48Z")
Yes, this is correct, eval is slow due to generation ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 15, 2020, 3:39pm UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282/4 "2020-07-15T15:39:57Z")
Is it possible to turn off generation but still get validation loss and save the best checkpoint based on that? There is no need for generation if I do not want ROUGE or BLEU scores during training, right? ---
**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\ **Post date:** [July 16, 2020, 11:04am UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282/5 "2020-07-16T11:04:28Z")
If you install from source, you can edit finetune.py to do as you describe šŸ™‚ The function you want is [here](https://github.com/huggingface/transformers/blob/aefc0c04294307dfe0eef42b607d4c877e72ed5b/examples/seq2seq/finetune.py#L164), and you could introduce a switch to only generate if called from `test_step`. Validation loss is also calculated within this function and is the basis upon which checkpoints are ranked (as far as I understand) ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 16, 2020, 11:12am UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282/6 "2020-07-16T11:12:30Z")
I think checkpoints are selected based on `val_metric` which can be anything I write code for (currently ROUGE and BLEU are implemented, I already added Levenshtein distance but that also requires generation) ---
**Author:** ![jenniferL](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jenniferl/32/33781_2.png) [@jenniferL](https://discuss.huggingface.co/u/jenniferL)\ **Post date:** [February 26, 2022, 10:38pm UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282/7 "2022-02-26T22:38:07Z")
Hi @valhalla @chrisdoyleIE, Thank you very much for answering the question. Could you please explain a little more about what ā€˜generationā€™ mean under this context? Personally, I feel like training still involves generation. In a forward pass, after encoder passes context vector to decoder, the decoder will still have to ā€˜generateā€™ a sequence. And during the generation of a sequence, for each step, the (cross entropy) loss function can calculate the difference between the softmax-ed distribution of all tokens with the ground truth distribution. If it still requires the whole predicted sequence to be ā€˜generatedā€™ in order to calculate loss, why would the answer be that ā€˜generationā€™ is solely performed for validation and that is the reason for it to be time consuming? I appreciate any feedback on this. Thanks in advance! Hi @chrisdoyleIE, Could you please elaborate more on how teacher forcing can speed up the training process? Thank you! ---
**Author:** ![anujsahani01](https://avatars.discourse-cdn.com/v4/letter/a/35a633/32.png) [@anujsahani01](https://discuss.huggingface.co/u/anujsahani01)\ **Post date:** [June 20, 2023, 11:51pm UTC](https://discuss.huggingface.co/t/seq2seq-evaluation-speed-is-slow/282/8 "2023-06-20T23:51:42Z")
Using Evaluators is a better and faster option, there are different evaluators for different tasks. from evaluate import TranslationEvaluator metric = TranslationEvaluator(task = ā€˜translationā€™, default\_metric\_name = ā€œbleuā€) evaluation\_results = metric.compute(model\_or\_pipeline = model, tokenizer = tokenizer, metric = ā€˜blueā€™, data = dataset[ā€˜testā€™] , device = 0, split = ā€˜testā€™ ,input\_column = ā€˜inputsā€™, label\_column = ā€˜targetsā€™) I used this for my translation model. One thing to note: The dataset we pass here should be normal text data, not tokenized data. refer this article > **[Evaluator](https://huggingface.co/docs/evaluate/package_reference/evaluator_classes#evaluate.TranslationEvaluator)** > > Weā€™re on a journey to advance and democratize artificial intelligence through open source and open science.