# Logging & Experiment tracking with W&B **URL:** \ **Category:** 🤗Transformers\ **Created:** [July 28, 2020, 12:12am UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498 "2020-07-28T00:12:30Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [July 28, 2020, 12:12am UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/1 "2020-07-28T00:12:30Z")
For people interested in tools for logging and comparing different models and training runs in general, [Weights & Biases](https://docs.wandb.com/library/integrations/huggingface) is directly integrated with 🤗 Transformers. You just need to have `wandb` installed and logged in. It automatically logs losses, metrics, learning rate, computer ressources, etc. ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/5/5ca3f47a5d1a6963c0ea36ad25627ef2929c6a2f.png) Here is another cool [example](https://app.wandb.ai/wandb/huggingtweets/sweeps/g5hwbevb) where I ran a sweep to fine-tune GPT-2 on tweets. ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/9d713e17a42758e5ccc393fb736ef574e736be53.png) Finally you can use your runs to create cool reports. See for example my [huggingtweets report](https://app.wandb.ai/wandb/huggingtweets/reports/HuggingTweets-Train-a-model-to-generate-tweets--VmlldzoxMTY5MjI). See [documentation](https://docs.wandb.com/library/integrations/huggingface) for more details or [this colab](http://wandb.me/hf). At the moment it is integrated with `Trainer` and `TFTrainer`. If you use Pytorch Lightning, you can use `WandbLogger`. See [Pytorch Lightning documentation](https://docs.wandb.com/library/integrations/lightning). Let me know if you have any questions or ideas to make it better! ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [September 1, 2020, 2:44am UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/2 "2020-09-01T02:44:20Z")
@boris I have a few questions for the HF Transformers integration: 1. It looks like `wandb` is charting the loss, learning rate, and epoch for a given run of `Trainer.train()`. Are there other things that would be useful to have charted for a finetuning run? 2. It also looks like `wandb` is using the `logging_steps` value in `TrainerArguments`. Is this right? 3. Is it preferred to set `wandb` behavior through the environment variables or in the finetuning script directly? ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [September 2, 2020, 4:11pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/3 "2020-09-02T16:11:39Z")
Hi @aclifton314, > [@aclifton314](#): > > 1. It looks like `wandb` is charting the loss, learning rate, and epoch for a given run of `Trainer.train()` . Are there other things that would be useful to have charted for a finetuning run? It also logs validation loss and all the task dependent metrics defined against your validation dataset. Then you can decide to log losses/metrics against epoch using it as x-axis in W&B interface (instead of the step) if it makes more sense for your use case. > [@aclifton314](#): > > 1. It also looks like `wandb` is using the `logging_steps` value in `TrainerArguments` . Is this right? That is correct. It can also log evaluation loss/metrics at the end of training if the `evaluate` loop is called at the end of your script (usually the case when you use one of the “examples” script from the library). > [@aclifton314](#): > > 1. Is it preferred to set `wandb` behavior through the environment variables or in the finetuning script directly? There is no preferred way. When I write my own fine-tuning script, I like to pass explicitly my variables as the code looks more clear to me but either way should work perfectly fine. ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [September 2, 2020, 7:28pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/4 "2020-09-02T19:28:29Z")
@boris. Thanks for your reply! One more quick question. I see where `wandb` gets initialized in the HF integration. Suppose I want to also log some other metric or value that isn’t automatically logged in the integration. Is there a way to call the same `wandb` object from my script that the HF inmplementation is using to include this value being logged? ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [September 2, 2020, 7:32pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/5 "2020-09-02T19:32:01Z")
Yes, you can just call `wandb.log` anytime manually like so `wandb.log({"my_metric":1.2}, step=trainer.global_step)` Passing the step is not mandatory but preferred when logging at different places. ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [September 3, 2020, 3:29pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/6 "2020-09-03T15:29:13Z")
@sgugger Can you make top post a wiki to update docs link and add relevant updates when needed? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [September 3, 2020, 4:07pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/7 "2020-09-03T16:07:46Z")
It’s done! ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [September 3, 2020, 4:25pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/8 "2020-09-03T16:25:18Z")
I’m thinking that we could add a way to log & track datasets and trained models. We could either do it with an environment variable or another parameter in `TrainingArguments`. ---
**Author:** ![katharinafluch](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@katharinafluch](https://discuss.huggingface.co/u/katharinafluch)\ **Post date:** [September 4, 2020, 9:35am UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/9 "2020-09-04T09:35:26Z")
@boris How did you manage to create such a sweep? Have you used Google Colab or any local machine? (I’m currently struggling with setting up one in Colab) In the docs it says that there are troubles by using wandb.agent with GPU-Support ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [September 4, 2020, 11:30pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/10 "2020-09-04T23:30:21Z")
@katharinafluch There are many ways to run sweeps but I actually ran mine in the console. There is a new version of wandb coming up soon that will better support sweeps in Colab so I can make a demo for it then! ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [September 28, 2020, 4:51pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/11 "2020-09-28T16:51:39Z")
@boris I have `export WANDB_MODE='dry_run'` and `WANDB_WATCH='all'` setup in my environment. I ran a training session and synced the dryrun using `wandb local`. When I view the results on the localhost, I don’t see any information about the parameters. Just plots about learning rate, epoch, and loss. Do you know where I can view this information about the parameters and/or if I have done something wrong to prevent them being recorded? ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [September 28, 2020, 5:10pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/12 "2020-09-28T17:10:11Z")
Is it a tensorflow model? I believe watch works mainly for Pytorch models. Also watch does not work with TPU. ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [September 28, 2020, 5:28pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/13 "2020-09-28T17:28:32Z")
It is a pytorch model and I am running on a single gpu. ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [September 28, 2020, 5:50pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/14 "2020-09-28T17:50:47Z")
Ok then maybe it’s because you have less than 100 training steps. Watch logs every 100 steps by default. Try training for more epochs. ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [September 28, 2020, 6:44pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/15 "2020-09-28T18:44:59Z")
I checked the `TrainArguments` object and it was set to 12,000 steps. ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [September 30, 2020, 2:56pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/16 "2020-09-30T14:56:27Z")
@boris, should the information about the parameters appear alongside the learning rate, epoch, and loss or are they somewhere else in the wandb dashboard? ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [September 30, 2020, 6:10pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/17 "2020-09-30T18:10:09Z")
@aclifton314 It appears in separate sections. ![wandb.ai_borisd13_huggingface_runs_1zdvv8or_workspace=user-borisd13](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/788e2f275a5882f68fd7322ae837786233993409.png) By default, you get gradients logged under “gradients” (as long as you have more than 100 training steps). You can also log parameters by setting `WANDB_WATCH` to 'all` and you will get both parameters and gradients (see [documentation](https://docs.wandb.com/library/integrations/huggingface)). I made a [demo colab](https://colab.research.google.com/drive/1tGKlzwlyhccF2ESjGZ6Qzxzrsr5SNlaW?usp=sharing) which also logs both gradients and parameters. Feel free if you have any other questions! ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [October 1, 2020, 6:04pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/18 "2020-10-01T18:04:11Z")
@boris oh awesome! Thank you for pointing that out. I had totally missed it. I am not seeing the gradients section, though. Is there something I need to modify in my setup? ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [October 1, 2020, 6:11pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/19 "2020-10-01T18:11:57Z")
Do they appear when you use my colab? It worked on [my run](https://wandb.ai/borisd13/huggingface-demo/runs/2idldto3?workspace=user-borisd13) You just need to have at least 100 training steps. ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [October 1, 2020, 6:17pm UTC](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/20 "2020-10-01T18:17:23Z")
I have `export WANDB_WATCH='all'` in my `.bashrc`. Do you think the formatting of that needs to be changed? [Next page](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498.md?page=2)