# Trainer.evaluate()

**URL:** <https://discuss.huggingface.co/t/trainer-evaluate/6107>\
**Category:** 🤗Transformers\
**Created:** [May 10, 2021, 8:18pm UTC](https://discuss.huggingface.co/t/trainer-evaluate/6107 "2021-05-10T20:18:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![pierreguillou](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pierreguillou/32/19626_2.png) [@pierreguillou](https://discuss.huggingface.co/u/pierreguillou)\
**Post date:** [May 10, 2021, 8:18pm UTC](https://discuss.huggingface.co/t/trainer-evaluate/6107/1 "2021-05-10T20:18:46Z")

</div>

Hello,

When the following code is run several times (notebook [language\_modeling.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb)), it gives a diferent value at each time:

```
import math
eval_results = trainer.evaluate()
print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")

```

I do not understand why (the eval loss should be always the same when using the same eval dataset and the same model).

---

<div class="post-metadata">

**Author:** ![infinitejoy](https://avatars.discourse-cdn.com/v4/letter/i/ea666f/32.png) [@infinitejoy](https://discuss.huggingface.co/u/infinitejoy)\
**Post date:** [May 11, 2021, 12:05pm UTC](https://discuss.huggingface.co/t/trainer-evaluate/6107/2 "2021-05-11T12:05:11Z")

</div>

I dont think its the exact same model and the same evaluation dataset everytime. Can you actually save and check the eval dataset across two runs please

---

<div class="post-metadata">

**Author:** ![pierreguillou](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pierreguillou/32/19626_2.png) [@pierreguillou](https://discuss.huggingface.co/u/pierreguillou)\
**Post date:** [May 11, 2021, 12:59pm UTC](https://discuss.huggingface.co/t/trainer-evaluate/6107/3 "2021-05-11T12:59:40Z")

</div>

Hello @infinitejoy.

> [@infinitejoy](#):
>
> I dont think its the exact same model

How the model could not be the same?  
When you run `trainer.evaluate()`, the model used is this one: `trainer.model`.

Just run `trainer.evaluate()` 2 times in a row in the Colab notebook [language.modeling.ipynb](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) and you’ll see a different perplexity (… with the same model).

```
import math

eval_results = trainer.evaluate()
print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")

eval_results = trainer.evaluate()
print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")

```

> [@infinitejoy](#):
>
> I dont think its (…) the same evaluation dataset

Same comment: when you run 2 times the same code `trainer.evaluate()`, it is the same evaluation set defined at the beginning of the notebook by the following code:

```
from datasets import load_dataset
datasets = load_dataset('wikitext', 'wikitext-2-raw-v1')

```

Could you run the Colab notebook and publish your results? Thanks. (I did it for Masked LM, not for Causal LM)

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [May 11, 2021, 1:05pm UTC](https://discuss.huggingface.co/t/trainer-evaluate/6107/4 "2021-05-11T13:05:54Z")

</div>

For masked LM, the tokens are randomly masked each time you go through the dataloader (for training and evaluation) in this notebook. That is why you see slightly different results at each run.