# Plot Loss Curve with Trainer()

**URL:** <https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767>\
**Category:** Beginners\
**Created:** [September 7, 2021, 8:28am UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767 "2021-09-07T08:28:04Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 7, 2021, 8:28am UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/1 "2021-09-07T08:28:04Z")

</div>

Hey,

I am fine tuning a BERT model for a Multiclass Classification problem. While training my losses seem to look a bit “unhealthy” as my validation loss is always smaller (eval\_steps=20) than my training loss. How can I plot a loss curve with a Trainer() model?

---

<div class="post-metadata">

**Author:** ![scottire](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/scottire/32/2685_2.png) [@scottire](https://discuss.huggingface.co/u/scottire)\
**Post date:** [September 11, 2021, 7:58pm UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/2 "2021-09-11T19:58:36Z")

</div>

Scott from Weights & Biases here. Don’t want to be spammy so will delete this if it’s not helpful. You can plot losses to W&B by passing `report_to` to TrainingArguments.

```auto
from transformers import TrainingArguments, Trainer

args = TrainingArguments(... , report_to="wandb")
trainer = Trainer(... , args=args)

```

More info here: [Logging & Experiment tracking with W&B](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498)

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 13, 2021, 7:09am UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/3 "2021-09-13T07:09:06Z")

</div>

Hey Scott,  
I think its helpful but I already do that. Anyway I want to find a way to directly plot the Losses in my notebook… . Any idea how to achieve that? Cheers

---

<div class="post-metadata">

**Author:** ![Pablogps](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pablogps/32/32528_2.png) [@Pablogps](https://discuss.huggingface.co/u/Pablogps)\
**Post date:** [September 13, 2021, 11:52am UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/4 "2021-09-13T11:52:05Z")

</div>

Note that validation losses being smaller than train is not necessarily bad or weird when working with advanced architectures and techniques, since you are not really comparing equivalent things. For example, consider dropout, that “cancels” some connections at train, while using all during evaluation (validation).

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 13, 2021, 12:10pm UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/6 "2021-09-13T12:10:31Z")

</div>

I trained a few other Bert models and it seems that all models need a few steps (up to 50) till the train loss becomes lower compared to the validation loss. Even with different random states etc. Do you think I do not really have to worry? I mean after those “starting problems” the losses behave normal/healthy for my taste (0.3 vs 0.6 when finished with early stopping)

---

<div class="post-metadata">

**Author:** ![Pablogps](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pablogps/32/32528_2.png) [@Pablogps](https://discuss.huggingface.co/u/Pablogps)\
**Post date:** [September 13, 2021, 12:30pm UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/7 "2021-09-13T12:30:21Z")

</div>

I obviously can’t say! But the fact that val loss is lower than train would not be a big concern to me! How those losses evolve seems more important. And of course if the model performance actually improves with time, that’s also more relevant! (You can see this in downstream tasks if training a language model).

---

<div class="post-metadata">

**Author:** ![scottire](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/scottire/32/2685_2.png) [@scottire](https://discuss.huggingface.co/u/scottire)\
**Post date:** [September 13, 2021, 2:08pm UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/8 "2021-09-13T14:08:12Z")

</div>

You should be able to use `%%wandb` at the beginning of your training loop cell to see the live graphs in the output.  
See: [Tracking Jupyter Notebooks](https://docs.wandb.ai/guides/track/jupyter)

---

<div class="post-metadata">

**Author:** ![gooohjy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gooohjy/32/30212_2.png) [@gooohjy](https://discuss.huggingface.co/u/gooohjy)\
**Post date:** [November 10, 2021, 10:54am UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/9 "2021-11-10T10:54:28Z")

</div>

Hey scottire, is it possible for me to obtain the training metrics and load them into a pandas dataframe? I’m looking to plot these scores in matplotlib so that I can compare with models trained with other frameworks.

Also, for using wandb is there a way for me to view the plot against epochs rather than steps?

Cheers!

---

<div class="post-metadata">

**Author:** ![scottire](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/scottire/32/2685_2.png) [@scottire](https://discuss.huggingface.co/u/scottire)\
**Post date:** [November 23, 2021, 9:29pm UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/10 "2021-11-23T21:29:14Z")

</div>

Yep that’s possible with the wandb API, see here: [https://docs.wandb.ai/guides/track/public-api-guide#querying-multiple-runs](https://docs.wandb.ai/guides/track/public-api-guide#querying-multiple-runs)

```auto

import pandas as pd 
import wandb

api = wandb.Api()
entity, project = "<entity>", "<project>" # set to your entity and project 
runs = api.runs(entity + "/" + project) 

summary_list, config_list, name_list = [], [], []
for run in runs: 
    # .summary contains the output keys/values for metrics like accuracy.
    # We call ._json_dict to omit large files 
    summary_list.append(run.summary._json_dict)

    # .config contains the hyperparameters.
    # We remove special values that start with _.
    config_list.append(
        {k: v for k,v in run.config.items()
         if not k.startswith('_')})

    # .name is the human-readable name of the run.
    name_list.append(run.name)

runs_df = pd.DataFrame({
    "summary": summary_list,
    "config": config_list,
    "name": name_list
    })

```

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [November 24, 2021, 8:23am UTC](https://discuss.huggingface.co/t/plot-loss-curve-with-trainer/9767/11 "2021-11-24T08:23:46Z")

</div>

Super cool! Thank you so much