# Trainer.evaluate() vs trainer.predict() **URL:** \ **Category:** 🤗Transformers\ **Created:** [December 19, 2022, 5:24pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-vs-trainer-predict/28132 "2022-12-19T17:24:04Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![curtmiller22](https://avatars.discourse-cdn.com/v4/letter/c/b487fb/32.png) [@curtmiller22](https://discuss.huggingface.co/u/curtmiller22)\ **Post date:** [December 19, 2022, 5:24pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-vs-trainer-predict/28132/1 "2022-12-19T17:24:04Z")
I am following the multilabel text classification tutorial from @nielsr located here: [Transformers-Tutorials/Fine\_tuning\_BERT\_(and\_friends)\_for\_multi\_label\_text\_classification.ipynb at master · NielsRogge/Transformers-Tutorials · GitHub](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb) I currently have my dataset split into a train, test, and validation dataset. After training, trainer.evaluate() is called which I think is being done on the validation dataset. My question is how do I use the model I created to predict the labels on my test dataset? Do I just call trainer.predict() immediately after trainer.evaluate() like so? trainer = Trainer( model, args, train\_dataset=encoded\_dataset[“train”], eval\_dataset=encoded\_dataset[“validation”], tokenizer=tokenizer, compute\_metrics=compute\_metrics ) trainer.evaluate() trainer.predict(encoded\_dataset[“test”]) Or can I just skip trainer.evaluate() and immediately go to trainer.predict() like so? trainer = Trainer( model, args, train\_dataset=encoded\_dataset[“train”], eval\_dataset=encoded\_dataset[“validation”], tokenizer=tokenizer, compute\_metrics=compute\_metrics ) trainer.predict(encoded\_dataset[“test”]) Any help would be greatly appreciated. Thank you! ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [December 20, 2022, 3:52pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-vs-trainer-predict/28132/2 "2022-12-20T15:52:43Z")
It depends on what you’d like to do, [trainer.evaluate()](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.evaluate) will predict + compute metrics on your test set and [trainer.predict()](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.predict) will only predict labels on your test set. However in case the test set also contains ground-truth labels, the latter will also compute metrics. ---
**Author:** ![curtmiller22](https://avatars.discourse-cdn.com/v4/letter/c/b487fb/32.png) [@curtmiller22](https://discuss.huggingface.co/u/curtmiller22)\ **Post date:** [December 21, 2022, 10:31pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-vs-trainer-predict/28132/3 "2022-12-21T22:31:07Z")
Thanks for getting back to me. Maybe my question is more related to what’s happening in inside trainer.train() and the difference between validation and prediction. After every training epoch (at least the way it is set up in the tutorial notebook), isn’t the model being evaluated against the validation dataset? So why is trainer.evaluate() being run on the validation dataset? Wouldn’t you want it to be the test dataset? ---
**Author:** ![Alegzandra](https://avatars.discourse-cdn.com/v4/letter/a/3da27b/32.png) [@Alegzandra](https://discuss.huggingface.co/u/Alegzandra)\ **Post date:** [February 28, 2023, 2:16pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-vs-trainer-predict/28132/4 "2023-02-28T14:16:05Z")
Hi! I have encountered the same problem, when running the same notebook… Did you manage to find the answer?.. ---
**Author:** ![Alegzandra](https://avatars.discourse-cdn.com/v4/letter/a/3da27b/32.png) [@Alegzandra](https://discuss.huggingface.co/u/Alegzandra)\ **Post date:** [March 15, 2023, 3:16pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-vs-trainer-predict/28132/5 "2023-03-15T15:16:29Z")
I solved the problem by specifying the test dataset as evaluation parameter: trainer.evaluate(eval\_dataset=encoded\_dataset[“test”]) ---
**Author:** ![Fardan](https://avatars.discourse-cdn.com/v4/letter/f/9fc29f/32.png) [@Fardan](https://discuss.huggingface.co/u/Fardan)\ **Post date:** [February 24, 2024, 5:54pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-vs-trainer-predict/28132/6 "2024-02-24T17:54:17Z")
You can set your test dataset as eval dataset on the fly post training completion. trainer.eval\_dataset = then run trainer.evaluate() ---
**Author:** ![taehyunzzz](https://avatars.discourse-cdn.com/v4/letter/t/ec9cab/32.png) [@taehyunzzz](https://discuss.huggingface.co/u/taehyunzzz)\ **Post date:** [July 10, 2024, 5:49am UTC](https://discuss.huggingface.co/t/trainer-evaluate-vs-trainer-predict/28132/7 "2024-07-10T05:49:35Z")
I might have a clue for generative tasks. I’ve had the same problem for summarization tasks, but it seems that generation length for evaluation mode is longer than that of validation, which answers why the two accuracies had been different. You should check the evaluation kwargs to see the differneces!