# Continuing Pre Training from Model Checkpoint **URL:** \ **Category:** Models\ **Created:** [November 5, 2021, 2:34pm UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392 "2021-11-05T14:34:18Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![fozyurt](https://avatars.discourse-cdn.com/v4/letter/f/c68b51/32.png) [@fozyurt](https://discuss.huggingface.co/u/fozyurt)\ **Post date:** [November 5, 2021, 2:34pm UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/1 "2021-11-05T14:34:18Z")
Hi, I pre-trained a language model for my own data and I want to continue the pre-training for additional steps using the last checkpoint. I am planning to use the code below to continue the pre-training but want to be sure that everything is correct before starting. Let’s say that I saved all of my files into CRoBERTa. model = RobertaForMaskedLM.from\_pretrained(‘CRoBERTa/checkpoint-…’) tokenizer = RobertaTokenizerFast.from\_pretrained(‘CRoBERTa’, max\_len = 512, padding = ‘longest’) training\_args = TrainingArguments(overwrite\_output\_dir = False, …) trainer = Trainer(…) trainer.train(resume\_from\_checkpoint = True) Is this pipeline correct ? Is there anything I am missing ? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [November 5, 2021, 2:39pm UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/2 "2021-11-05T14:39:21Z")
If you use ```auto trainer.train(resume_from_checkpoint = True) ``` The Trainer will load the last checkpoint it can find, so it won’t necessarily be the one you specified. It will also resume the training from there with just the number of steps left, so it won’t be any different from the model you got at the end of your initial `Trainer.train`. ---
**Author:** ![fozyurt](https://avatars.discourse-cdn.com/v4/letter/f/c68b51/32.png) [@fozyurt](https://discuss.huggingface.co/u/fozyurt)\ **Post date:** [November 5, 2021, 2:41pm UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/3 "2021-11-05T14:41:29Z")
So in this case I don’t need to specify the checkpoint when loading the pre-trained model and the rest is good to go, right ? ---
**Author:** ![FarisHijazi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/farishijazi/32/3501_2.png) [@FarisHijazi](https://discuss.huggingface.co/u/FarisHijazi)\ **Post date:** [August 9, 2022, 2:51pm UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/4 "2022-08-09T14:51:38Z")
ok I’ve tried ```auto trainer.train(resume_from_checkpoint = True) ``` and it does load and train successfully, but when I check my logger (eg tensorboard), every time I train the epochs start from 0, and it’s annoying because the curves keep starting from the beginning when they should actually be back-to-back am I doing something wrong? and is there a way to fix this? ---
**Author:** ![FarisHijazi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/farishijazi/32/3501_2.png) [@FarisHijazi](https://discuss.huggingface.co/u/FarisHijazi)\ **Post date:** [August 9, 2022, 2:54pm UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/5 "2022-08-09T14:54:17Z")
I have one more questions which is unrelated and I found no answers for, is how do I save model checkpoints in the same format as `trainer.train()` does? I know I can use `model.save_pretrained('bert-base-uncased')`, however this saves directly in that directory, unlike `trainer.train()` which saves in `bert-base-uncased/checkpoint-100` …, I want a function that will automatically do this based on the current step count, does such a function exist? ---
**Author:** ![canovich](https://avatars.discourse-cdn.com/v4/letter/c/aeb1de/32.png) [@canovich](https://discuss.huggingface.co/u/canovich)\ **Post date:** [August 30, 2022, 3:21am UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/6 "2022-08-30T03:21:25Z")
I know that we can continue training with this: ```auto trainer.train("checkpoint-9500") ``` But for example, i wanted to change part of a model (for example wanted to change embedding size of model.) How can i do that? Is it possible? ---
**Author:** ![thasan](https://avatars.discourse-cdn.com/v4/letter/t/df705f/32.png) [@thasan](https://discuss.huggingface.co/u/thasan)\ **Post date:** [November 5, 2023, 9:12am UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/7 "2023-11-05T09:12:06Z")
did you able to solve it? ---
**Author:** ![CyrisHe](https://avatars.discourse-cdn.com/v4/letter/c/d78d45/32.png) [@CyrisHe](https://discuss.huggingface.co/u/CyrisHe)\ **Post date:** [December 15, 2023, 8:51am UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/8 "2023-12-15T08:51:03Z")
> [@FarisHijazi](#): > > it’s annoying because the curves keep starting from the beginning when they should actually be back-to-back Yes, I met the same situation. For example, the loss at checkpoint-100 was 0.20, but when I set resume from checkpoit to True, the training still start from step 1 and the loss is still 0.7, just as I was start from the beginning. ---
**Author:** ![stevenaaa1207](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@stevenaaa1207](https://discuss.huggingface.co/u/stevenaaa1207)\ **Post date:** [January 28, 2024, 2:12am UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/9 "2024-01-28T02:12:13Z")
i just downloaded my checkpoint files from “checkpoint510” and upload it to another machine(same service in vast.ai) , also the resume from check point setup. But there is still a error of no valid checkpoint plz someone tell me why. ---
**Author:** ![ghashami](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ghashami/32/24400_2.png) [@ghashami](https://discuss.huggingface.co/u/ghashami)\ **Post date:** [March 2, 2024, 11:19pm UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/10 "2024-03-02T23:19:00Z")
I am able to train from the best checkpoint but the model does not write down new checkpoints after training completes. why? ---
**Author:** ![k253](https://avatars.discourse-cdn.com/v4/letter/k/aca169/32.png) [@k253](https://discuss.huggingface.co/u/k253)\ **Post date:** [April 16, 2024, 6:57am UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/11 "2024-04-16T06:57:19Z")
I meet a problem about mismatch in loss when using torch.load and resume\_from\_checkpoint, here my detailed problem description. [python - Mismatch in Loss When Using `torch.load` and `resume_from_checkpoint` - Stack Overflow](https://stackoverflow.com/questions/78332638/mismatch-in-loss-when-using-torch-load-and-resume-from-checkpoint). Expecting your reply. ---
**Author:** ![dlface](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dlface/32/27767_2.png) [@dlface](https://discuss.huggingface.co/u/dlface)\ **Post date:** [June 4, 2024, 3:20am UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/12 "2024-06-04T03:20:29Z")
I meet the same problem. resume trainning works, but train loss and eval loss both increased. ---
**Author:** ![qswadey](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/qswadey/32/39080_2.png) [@qswadey](https://discuss.huggingface.co/u/qswadey)\ **Post date:** [January 13, 2025, 11:36am UTC](https://discuss.huggingface.co/t/continuing-pre-training-from-model-checkpoint/11392/13 "2025-01-13T11:36:39Z")
**If our trainer is defined as:** my\_training\_args = TrainingArguments( report\_to=“none”, output\_dir=“./expt2\_train\_student\_gpt”, num\_train\_epochs=100, save\_strategy = “epoch”, … … push\_to\_hub=False, ) _# Create a trainer for evaluation_ scratch\_trainer = Trainer( model = scratch\_train\_model, args = my\_training\_args, … ) **and we initially started a 100-epoch training:** scratch\_trainer.train() **We can extend it to 200 epoch in the following manner:** **redefine the trainer as follows and run this cell again (note: it has the increased epochs now):** _my_\_training\_args = TrainingArguments( report\_to=“none”, output\_dir=“./expt2\_train\_student\_gpt”, num\_train\_epochs=200, save\_strategy = “epoch”, … … push\_to\_hub=False, ) _# Create a trainer for evaluation_ scratch\_trainer = Trainer( model = scratch\_train\_model, args = my\_training\_args, … ) **resume training here:** _# We are attempting a resume here_ scratch\_trainer.train(resume\_from\_checkpoint = True) I have found that this starts slightly above the halfway (say from 53rd epoch for the earlier 100 epoch training and the loss value is quite good - I mean it does not start from the beginning loss value, but starts from a lower loss eventually converges to the earlier loss level. Then it continues training to the new desired epochs. `In summary, we can recover around 50% of the training, unless there is a major change in training hyperparameters.`