# Seq2SeqTrainer Questions

**URL:** <https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276>\
**Category:** 🤗Transformers\
**Created:** [September 28, 2020, 3:06pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276 "2020-09-28T15:06:07Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 28, 2020, 3:06pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/1 "2020-09-28T15:06:07Z")

</div>

@valhalla

Questions about `Seq2SeqTrainer`

1. does it know how to do multigpu sortish sampler?
2. does it know how to sync metrics in a multigpu setting?
3. is TPU faster than GPU ?

### Experiments

1. Would be interested to know how finetune bart-large on xsum performs, for example, esp. on TPU .
2. Would be cool to make a blogpost showing some sort of optuna/ray search for summarization or translation finetuning. Would also be cool to know the best hparams.  
Good hparams to check are learning rate, gradient\_accumulation\_steps, label\_smoothing, dropout.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 28, 2020, 3:51pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/2 "2020-09-28T15:51:29Z")

</div>

1. Yes, it uses `DistributedSortishSampler` for multi-gpu, it’s handled [here](https://github.com/huggingface/transformers/blob/f62f2ffdcc2df75cf01438bebc7ae281d921d21d/examples/seq2seq/seq2seq_trainer.py#L31)

2. Yes, metrics are handled by `Trainer` and logged in the global main process. pinging @sgugger for confirmation. `TPU` training is multi-device, so for example you see TPU logs [here](https://wandb.ai/psuraj/s2s-trainer-marian/runs/2i2pn8cm?workspace=user-psuraj) for distilbart-cnn-12-3.

> (Note: as `generate` crashes/hangs on TPU we don’t calculate BLEU, ROUGE metrics on TPU)

1. Yes, I’ve observed 3-4x speed boost on TPU v3-8 compared to single V100

- distilbart-cnn-12-3, 1 epoch
  - on TPU V3-8: 25.5 min
  - on V100: ~102 min

- marin-enro-6-3 , 1 epoch
  - on TPU v3-8: ~5min
  - on V100 : 21 min

All experiments are logged in this [wandb project](https://wandb.ai/psuraj/s2s-trainer-marian) if you want to compare yourself .

1. Coming soon 😉

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 28, 2020, 11:52pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/3 "2020-09-28T23:52:47Z")

</div>

Yes metrics are properly accumulated and logged on the main process by `Trainer`.

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 29, 2020, 1:00am UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/4 "2020-09-29T01:00:19Z")

</div>

Awesome I will switch after I finish the distillbart paper! (PL cannot aggregate metrics to main project at the moment and TPU is broken). Should we deprecate/delete the PL trainer at some point?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 29, 2020, 1:08am UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/5 "2020-09-29T01:08:59Z")

</div>

When we have something that works well for Seq2Seq definitely.  
FYI, I’m full time on making Trainer better now that I’m mostly finished with the docs, so don’t hesitate to ask if you have any questions 🙂 I’m on the text classification examples to begin gently but will make my way up to seq2seq.

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 30, 2020, 1:25am UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/6 "2020-09-30T01:25:42Z")

</div>

@valhalla I tried to use today on multigpu and after fixing two bugs here is stream of consciousness from watching a marian training run:

1. For Seq2SeqTrainer,  
`--do_eval` is mandatory.  
Otherwise, you get `ValueError: Trainer: evaluation requires an eval_dataset.` at the end of epoch 1.  
If it’s mandatory the user should not need to specify it.

2. This may be more of an @sgugger question, but I don’t understand why I need to specify run name and output dir. Can we have run\_name = output\_dir if run\_name is not specified?

3. This stuff should all be rounded (and maybe lr and loss should just be part of the prog bar)

```auto
{'loss': 6397.63612890625, 'learning_rate': 0.0003, 'epoch': 0.10484378276368211, 'total_flos': 2023859842449408, 'step': 500}

```

Is total flos supposed to be total flops? Why would I want to see that?  
Isn’t it just step \* some constant?

Would be dope if these eval logs

```auto
{'eval_loss': 6391.572265625, 'eval_bleu': 25.9505, 'eval_gen_len': 33.37937937937938, 'epoch': 1.2581253931641854, 'total_flos': 23975950948909056, 'step': 6000}
{'eval_loss': 6391.572265625, 'eval_bleu': 25.9505, 'eval_gen_len': 33.37937937937938, 'epoch': 1.2581253931641854, 'total_flos': 24064800096190464, 'step': 6000}

```

Only got printed by rank 0 and also got written to disk by rank zero.

My progress bar says

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/9451f881835c26904d67a076b334fef4125dee32.png)  
After finishing 2/6 epochs.

Iteration does not clarify anything for me. I thought it was the whole job at the beginning. I wish that one would be called `f"Epoch {n}" Say “Train”  
I think I also have 2 progress bars, one from each proc.

1. log\_history.json is replicated inside every checkpoint dir? why?  
`dbg_distributed_mar/checkpoint-6000/log_history.json`

in the PL version you can always can output\_dir/metrics.json and see full, up to date metrics. You don’t have to `ls` first to find your most recent checkpoint dir and then `cat log_history.json`

1. We should be rounding stuff in log\_history.json

```auto
  {
    "loss": 4743.288, # YAY
    "learning_rate": 0.00020929785871807638,
    "epoch": 1.887188089746278,
    "total_flos": 36036642601893888,
    "step": 9000
  },
  {
    "eval_loss": 6336.4580078125,
    "eval_bleu": 26.2017,
    "eval_gen_len": 33.48048048048048,
    "epoch": 1.887188089746278,
    "total_flos": 36036642601893888,
    "step": 9000

```

I also think total\_flos is distracting and useless, but whatever.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 30, 2020, 2:20am UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/7 "2020-09-30T02:20:09Z")

</div>

1 . `--do_eval` shouldn’t be mandatory. If we set `--evaluate_during_training`  
without `--do_eval` then we see this issue. The bug is [here](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_trainer.py#L353) in `finetune_trainer.py`, it only loads `eval_dataset` when `--do_eval` is set, it should also check for  
`--evaluate_during_training` or the new `evaluation_strategy` argument.  
@sgugger I think this needs to be handled in every `example/` script, WDYT ?

4 . Progress bars in `Trainer` are bit different than PL, it creates one progress bar for each epoch and each eval loop (eval pbar is created for each device when on muli-gpu or TPU 🤯). And another one to track epochs.

5 . I’m also confused about why `log_history` is saved in every ckpt, @sgugger may know why.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 30, 2020, 1:44pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/8 "2020-09-30T13:44:52Z")

</div>

1. I didn’t write the part with `do_train`/`do_eval` and those arguments are not used by Trainer, only the scripts that use Trainer (not at the scripts yet). Maybe we can add something in the postinit of the TrainingArguments that if `do_eval` is not set it defaults to `evaluation_strategy != "no"`?

2. Yep, definitely possible.

3. logs are handled badly right now IMO cleanup is scheduled for later (see 4)

4. Progress bars are a mess in the Trainer right now, I usually run it with `disable_tqdm=True` for now, this is something I intend to clean after a PR I’m preparing with state + callbacks (since then progress bars can be handled nicely in a callback).

5. log\_history needs to be saved at each checkpoint to because it will be reloaded from there if you resume training. (They are not all duplicates each contain the history up until that point.)

6. Rounding should be done by the user IMO. I have no idea why `total_flos` are in the logs, you should ask @teven 🙂

---

<div class="post-metadata">

**Author:** ![teven](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/teven/32/120_2.png) [@teven](https://discuss.huggingface.co/u/teven)\
**Post date:** [September 30, 2020, 2:42pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/9 "2020-09-30T14:42:16Z")

</div>

Hey @sshleifer ! You’re right that flos are total operations; flops are operations per second so I thought it’d be clear that flos are operations, if you’ve got a better name I’m all ears since this hasn’t actually been that clear to everyone. You’re also right that for a given model, it is just step \* constant (at least nearly all the time). However, it is still interesting to compare models against each other, or different sizes of the same model, or the same model with different context sizes. Would you like it better if it was optional? I’ve been asked before in conference reviews for efficiency papers to compare floating-point operations, and it’s pretty integral to the `calculator` estimates, but I get that it might not be useful to everyone yet. (although I’m working on integrating `calculator` with Trainer on the side and should be done next week).

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 30, 2020, 2:46pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/10 "2020-09-30T14:46:17Z")

</div>

I’m working on cleaning the Trainer state. If total\_flos is not printed but inside a field of `Trainer.state` doest that work for you? Does it need to be logged at the end of training only maybe?

---

<div class="post-metadata">

**Author:** ![teven](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/teven/32/120_2.png) [@teven](https://discuss.huggingface.co/u/teven)\
**Post date:** [September 30, 2020, 3:02pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/11 "2020-09-30T15:02:19Z")

</div>

I don’t think it should be printed as it is not really human-readable information; but it is useful, for example, to graph the loss as function of the number of flos invested in training (not only the final sum), so I feel that it should be part of the logs.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 30, 2020, 4:11pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/12 "2020-09-30T16:11:56Z")

</div>

It depends on what we call logs since logs are printed 😉  
Do you want it sent to the integrations like TensorBoard/Comet/Wandb? Just accessible inside of the `Trainer` for some custom post-processing?

---

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [August 18, 2022, 3:07pm UTC](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/13 "2022-08-18T15:07:28Z")

</div>

> [@valhalla](#):
>
> All experiments are logged in this [wandb project](https://wandb.ai/psuraj/s2s-trainer-marian) if you want to compare yourself .
> 
> 1. Coming soon 😉

all my processes in wandb are logging so I get 8 runs ☹ How do I modify my trainer only code?

details: [Logging & Experiment tracking with W&B - #73 by brando](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498/73)

* * *

solution: [Is wandb in Trainer configured for distributed training? - #3 by brando](https://discuss.huggingface.co/t/is-wandb-in-trainer-configured-for-distributed-training/21067/3)