# No skipping steps after loading from checkpoint

**URL:** <https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880>\
**Category:** 🤗Transformers\
**Created:** [June 16, 2021, 7:04am UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880 "2021-06-16T07:04:58Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![stoffy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stoffy/32/1794_2.png) [@stoffy](https://discuss.huggingface.co/u/stoffy)\
**Post date:** [June 16, 2021, 7:04am UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/1 "2021-06-16T07:04:58Z")

</div>

Hey! I am trying to continue training by loading a checkpoint. But for some reason, it always starts from scratch. Probably I am just missing something.

```auto
training_arguments = Seq2SeqTrainingArguments(
            predict_with_generate=True,
            evaluation_strategy='steps',
            per_device_train_batch_size=training_config['per_device_train_batch_size'],
            per_device_eval_batch_size=training_config['per_device_eval_batch_size'],
            fp16=True,
            output_dir=training_output_path,
            overwrite_output_dir=True,
            logging_steps=training_config['logging_steps'],
            save_steps=training_config['save_steps'],
            eval_steps=training_config['eval_steps'],
            warmup_steps=training_config['warmup_steps'],
            metric_for_best_model='eval_loss',
            greater_is_better=False)

trainer = Seq2SeqTrainer(
            model=model,
            tokenizer=tokenizer,
            args=training_arguments,
            compute_metrics=compute_metrics,
            train_dataset=train_ds,
            eval_dataset=eval_ds,
        )

```

Here are the logs:

```auto
loading weights file .../models/checkpoint-2000/pytorch_model.bin
All model checkpoint weights were used when initializing EncoderDecoderModel.
*****Running training*****
  Num examples = 222862
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 83574

```

I am missing some like:

```auto
Continuing training from checkpoint, will skip to saved global_step
Continuing training from epoch 0
Continuing training from global step 48000
Continuing training from 0 non-embedding floating-point operations
Will skip the first 48000 steps in the first epoch

```

Which I found here: [Load from checkpoint not skipping steps - 🤗Transformers - Hugging Face Forums](https://discuss.huggingface.co/t/load-from-checkpoint-not-skipping-steps/1553/3)

Maybe somebody can help me? Thank you in advance!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [June 16, 2021, 12:59pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/2 "2021-06-16T12:59:59Z")

</div>

With `overwrite_output_dir=True` you reset the output dir of your `Trainer`, which deletes the checkpoints. If you remove that option, it should resume from the lastest checkpoint.

---

<div class="post-metadata">

**Author:** ![stoffy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stoffy/32/1794_2.png) [@stoffy](https://discuss.huggingface.co/u/stoffy)\
**Post date:** [June 17, 2021, 5:23am UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/3 "2021-06-17T05:23:22Z")

</div>

Thanks for your fast response. Unfortunately, it is still not working. I have set `overwrite_output_dir=False` but the outcome is the same:

```auto
loading weights file /content/drive/MyDrive/output/training/roberta/checkpoint-59000/pytorch_model.bin
All model checkpoint weights were used when initializing EncoderDecoderModel.

All the weights of EncoderDecoderModel were initialized from the model checkpoint at /content/drive/MyDrive/output/training/roberta/checkpoint-59000.
If your task is similar to the task the model of the checkpoint was trained on, you can already use EncoderDecoderModel for predictions without further training.
PyTorch: setting up devices
Using amp fp16 backend
*****Running training*****
  Num examples = 222862
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 83574
  0% 50/83574 [00:20<9:30:55, 2.44it/s]

```

Probably I don’t understand something here. When resuming I pick the checkpoint path as the model path. That’s correct right?

I am a bit confused by the documentation:

> **overwrite\_output\_dir** ( `bool` , optional, defaults to `False` ) – If `True` , overwrite the content of the output directory. Use this to continue training if `output_dir` points to a checkpoint directory.

Since I point to a checkpoint directory this should be set to `True`, right?

Sorry for so many questions. This is all very new to me.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [June 17, 2021, 11:36am UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/4 "2021-06-17T11:36:40Z")

</div>

Oh the documentation is outdated, you shouldn’t use your model from the checkpoint directory anymore, as long as the checkpoint is in the `output_dir`, the `Trainer` will use it if you do `trainer.train(resume_from_checkpoint=True)`.

You can also pass the folder to your exact checkpoint instead of `True`.

---

<div class="post-metadata">

**Author:** ![stoffy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stoffy/32/1794_2.png) [@stoffy](https://discuss.huggingface.co/u/stoffy)\
**Post date:** [June 17, 2021, 6:30pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/5 "2021-06-17T18:30:54Z")

</div>

Thanks a lot. It works like charm!

---

<div class="post-metadata">

**Author:** ![ThomasG](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomasg/32/2445_2.png) [@ThomasG](https://discuss.huggingface.co/u/ThomasG)\
**Post date:** [October 1, 2021, 11:40am UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/7 "2021-10-01T11:40:23Z")

</div>

Hello, can you elaborate on what you mean by saying “you shouldn’t use your model from the checkpoint directory anymore” ? If I want to continue training from my last checkpoint, how should I load my model using `from_pretrained()`?

Should I not pass the path to the checkpoint inside the `from_pretrained()` method?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 1, 2021, 11:57am UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/8 "2021-10-01T11:57:32Z")

</div>

No, you should just do:

```auto
trainer.train(resume_from_checkpoint=True)

```

as I said earlier. It will load the weights in your model.

---

<div class="post-metadata">

**Author:** ![ThomasG](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomasg/32/2445_2.png) [@ThomasG](https://discuss.huggingface.co/u/ThomasG)\
**Post date:** [October 1, 2021, 12:10pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/9 "2021-10-01T12:10:27Z")

</div>

Thanks for the very fast reply. I don’t understand how I must load my model in this case? I am using the `Wav2Vec2ForCTC` model, and I’ve fine-tuned it for 27 epochs, while initially I asked it to train for 100 epochs. But I had to stop it, and I have the checkpoint of the 27th epoch.

Am I supposed to load it using the initial `model = Wav2Vec2ForCTC.from_pretrained('facebook/wav2vec2-large-xlsr-53')` ?

If I do this, and then re-define the `trainer` & `training arguments`, then start training using the code you provided, will it over-write the pretrained loaded weights and use my saved ones?

Thanks!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 1, 2021, 12:13pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/10 "2021-10-01T12:13:14Z")

</div>

You should use your model the same way you did before, as I said the `trainer` will load the weights from the last checkpoint.

---

<div class="post-metadata">

**Author:** ![ThomasG](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomasg/32/2445_2.png) [@ThomasG](https://discuss.huggingface.co/u/ThomasG)\
**Post date:** [October 1, 2021, 12:23pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/11 "2021-10-01T12:23:41Z")

</div>

Thank you. Please forgive my ignorance, but just to make sure I understand everything correctly, the steps are as follows:

1. Load the model (using the typical from\_pretrained I showed above) and the trainer/arguments as before.
2. If the output\_dir of my arguments contains the checkpoint, use `overwrite_output_dir=True`.
3. Use `trainer.train(resume_from_checkpoint=True)`

This will continue training the model for the remainder of the epochs defined in my arguments, and will load the weights of my 27th epoch.

Does everything sound correct?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 1, 2021, 1:06pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/12 "2021-10-01T13:06:06Z")

</div>

No you should absolutely not use `overwrite_output_dir`, as it will overwrite on your checkpoints.

---

<div class="post-metadata">

**Author:** ![ThomasG](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomasg/32/2445_2.png) [@ThomasG](https://discuss.huggingface.co/u/ThomasG)\
**Post date:** [October 1, 2021, 1:10pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/13 "2021-10-01T13:10:08Z")

</div>

> [@stoffy](#):
>
> **overwrite\_output\_dir** ( `bool` , optional, defaults to `False` ) – If `True` , overwrite the content of the output directory. Use this to continue training if `output_dir` points to a checkpoint directory.

This is from the documentation. Isn’t this what I am doing? When should I set this to True? I am sorry for the many questions, it’s just that I am not sure what is the proper order to continue from a checkpoint, as there are many different choices to take into account.

In particular, when I re-define my trainer and training arguments, if my `output_dir` is the one containing the checkpoint, shouldn’t I set the `overwrite` to True? Can I even set a completely different `output_dir`, that does not contain my final checkpoint?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 1, 2021, 1:25pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/14 "2021-10-01T13:25:53Z")

</div>

The documentation is wrong in this case (it is very old so I’m guessins we forgot to update it). In any case:

- `overwrite_output_dir` is only used in the example scripts, not the `Trainer` class iself, so its value is irrelevant if you are not using an example script
- when using an example script it needs to be set to `False` to resume from a checkpoint.

---

<div class="post-metadata">

**Author:** ![ThomasG](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomasg/32/2445_2.png) [@ThomasG](https://discuss.huggingface.co/u/ThomasG)\
**Post date:** [October 1, 2021, 1:34pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/15 "2021-10-01T13:34:44Z")

</div>

I see. Thanks a lot @sgugger. In any case before I attempt anything, I will create a copy of my checkpoint just to be safe! 😃

Have a great day!

---

<div class="post-metadata">

**Author:** ![viv-om](https://avatars.discourse-cdn.com/v4/letter/v/43a26b/32.png) [@viv-om](https://discuss.huggingface.co/u/viv-om)\
**Post date:** [February 24, 2022, 10:45am UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/16 "2022-02-24T10:45:52Z")

</div>

@sgugger I am using trainer.train(resume\_from\_checkpoint=True) to train the model from last checkpoint but it starts from the beginning. I can see the checkpoints saved in the correct folder. I did earlier have overwrite\_output\_dir=True in my training args. I have removed it now but no avail.

Can you please comment on what could be going wrong here?

---

<div class="post-metadata">

**Author:** ![lucadini](https://avatars.discourse-cdn.com/v4/letter/l/d9b06d/32.png) [@lucadini](https://discuss.huggingface.co/u/lucadini)\
**Post date:** [March 3, 2022, 9:37am UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/17 "2022-03-03T09:37:11Z")

</div>

Maybe overwrite\_output\_dir=True deleted all your checkpoints, check the output directory! If this happened, you have to repeat the first part of the training and then execute the second part always keeping overwrite\_output\_dir=False.

---

<div class="post-metadata">

**Author:** ![dashapyly](https://avatars.discourse-cdn.com/v4/letter/d/87869e/32.png) [@dashapyly](https://discuss.huggingface.co/u/dashapyly)\
**Post date:** [April 21, 2022, 3:52pm UTC](https://discuss.huggingface.co/t/no-skipping-steps-after-loading-from-checkpoint/6880/19 "2022-04-21T15:52:32Z")

</div>

Actually, I have restarted everything, and now everything works fine and datapoints are being skipped - ignore me please. Thanks!