# Save only best model in Trainer **URL:** \ **Category:** 🤗Transformers\ **Created:** [July 17, 2021, 6:25am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442 "2021-07-17T06:25:08Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![prachi12](https://avatars.discourse-cdn.com/v4/letter/p/b2d939/32.png) [@prachi12](https://discuss.huggingface.co/u/prachi12)\ **Post date:** [July 17, 2021, 6:25am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/1 "2021-07-17T06:25:08Z")
I have read previous posts on the similar topic but could not conclude if there is a workaround to get only the best model saved and not the checkpoint at every step, my disk space goes full even after I add **savetotallimit** as 5 as the trainer saves every checkpoint to disk at the start. Please suggest. Thanks ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [July 17, 2021, 8:05am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/2 "2021-07-17T08:05:29Z")
You can set `save_strategy` to NO to avoid saving anything and save the final model once training is done with `trainer.save_model()`. ---
**Author:** ![prachi12](https://avatars.discourse-cdn.com/v4/letter/p/b2d939/32.png) [@prachi12](https://discuss.huggingface.co/u/prachi12)\ **Post date:** [July 17, 2021, 10:27am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/3 "2021-07-17T10:27:53Z")
Thank you, this is helpful. ---
**Author:** ![prachi12](https://avatars.discourse-cdn.com/v4/letter/p/b2d939/32.png) [@prachi12](https://discuss.huggingface.co/u/prachi12)\ **Post date:** [July 17, 2021, 4:22pm UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/4 "2021-07-17T16:22:43Z")
I am using the below - ```auto args = TrainingArguments( output_dir=f"./out_fold{i}", overwrite_output_dir = 'True', evaluation_strategy="steps", eval_steps=40, logging_steps = 40, learning_rate = 5e-5, per_device_train_batch_size=8, per_device_eval_batch_size=8, num_train_epochs=10, seed=0, save_total_limit = 1, # report_to = "none", # logging_steps = 'epoch', load_best_model_at_end=True, save_strategy = "no" ) trainer = Trainer( model=model, args=args, train_dataset=train_dataset, eval_dataset=val_dataset, # compute_metrics=compute_metrics, # callbacks=[EarlyStoppingCallback(early_stopping_pa)], ) trainer.train() trainer.save_model(f'out_fold{i}') ``` Here, thought save\_strategy = “no” , the checkpoints are saved at start in disk (as below) due to which disk goes full. Can you suggest what’s going wrong? \*\*\*\*\* Running Evaluation \*\*\*\*\* Num examples = 567 Batch size = 8 Saving model checkpoint to ./out\_fold0/checkpoint-40 Configuration saved in ./out\_fold0/checkpoint-40/config.json Model weights saved in ./out\_fold0/checkpoint-40/pytorch\_model.bin Deleting older checkpoint [out\_fold0/checkpoint-760] due to args.save\_total\_limit \*\*\*\*\* Running Evaluation \*\*\*\*\* ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [July 19, 2021, 9:21am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/5 "2021-07-19T09:21:32Z")
You can’t use `load_best_model_at_end=True` if you don’t want to save checkpoints: it needs to save checkpoints at every evaluation to make sure you have the best model, and it will always save 2 checkpoints (even if `save_total_limit` is 1): the best one and the last one (to resume an interrupted training). ---
**Author:** ![carted-ml](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/carted-ml/32/4381_2.png) [@carted-ml](https://discuss.huggingface.co/u/carted-ml)\ **Post date:** [March 30, 2022, 10:14am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/6 "2022-03-30T10:14:17Z")
If `save_total_limit` is set to some value, will checkpoints be replaced by newer checkpoints even if the newer checkpoints are underperforming? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [March 30, 2022, 2:46pm UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/7 "2022-03-30T14:46:02Z")
The best checkpoint is always kept, as is the last checkpoint (to make sure you can resume training from it). ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [March 30, 2022, 6:15pm UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/8 "2022-03-30T18:15:02Z")
> [@prachi12](#): > > `save_total_limit = 1,` Thanks @sgugger I believe we have to set these parameters which will save 2 checkpoints (best one and last one) and to avoid saving checkpoints at every evaluation. Is that right? save\_total\_limit = 2 save\_strategy = “no” load\_best\_model\_at\_end=False ---
**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\ **Post date:** [May 7, 2022, 10:39pm UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/9 "2022-05-07T22:39:01Z")
@Vinayaks117, did those settings work for you to save the most recent and the best? I’d like to do the same. ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [May 8, 2022, 7:55am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/10 "2022-05-08T07:55:45Z")
@jbmaxwell Yes. ---
**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\ **Post date:** [May 8, 2022, 3:34pm UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/11 "2022-05-08T15:34:26Z")
Great, thanks! ---
**Author:** ![ahadda5](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@ahadda5](https://discuss.huggingface.co/u/ahadda5)\ **Post date:** [July 12, 2022, 8:58am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/12 "2022-07-12T08:58:57Z")
tbh it didn’t work for me, (version 4.20.1) not sure what i’m missing. So instead i’m running a cron job every 15 min to clean up those checkpoints. file handlers limit gets reached after 10hrs ,so as a last resort the cron job is cleaning those files ---
**Author:** ![ahadda5](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@ahadda5](https://discuss.huggingface.co/u/ahadda5)\ **Post date:** [July 12, 2022, 1:09pm UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/13 "2022-07-12T13:09:21Z")
My bad it works, it was an issue with how i passed params through sys.argv ---
**Author:** ![artificial-cerebrum](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@artificial-cerebrum](https://discuss.huggingface.co/u/artificial-cerebrum)\ **Post date:** [September 12, 2022, 9:15am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/14 "2022-09-12T09:15:51Z")
@sgugger @prachi12 how do you know which checkpoint had the best performance and how do you load that specific checkpoint? ---
**Author:** ![astariul](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@astariul](https://discuss.huggingface.co/u/astariul)\ **Post date:** [November 15, 2022, 3:49am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/15 "2022-11-15T03:49:09Z")
@artificial-cerebrum I had the same question, I couldn’t find the answer from the documentation. After checking the source code, I found : ```python # Define your trainer, etc... trainer.train() # After training, access the path of the best checkpoint like this best_ckpt_path = trainer.state.best_model_checkpoint ``` ---
**Author:** ![artificial-cerebrum](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@artificial-cerebrum](https://discuss.huggingface.co/u/artificial-cerebrum)\ **Post date:** [December 11, 2022, 2:58pm UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/16 "2022-12-11T14:58:27Z")
@astariul - Thanks! That seems to be working. For future readers, I think another option would be to remove the whole directory (`rm -r $save_path`) after training, and then do `trainer.save_model()`. It seems that this way it saves only the best model (assuming you had enabled `load_best_model=True`). Alternatively, if you don’t want to delete the checkpoints, then you can avoid `rm -r $save_path`, and provide a new `output_dir` path to `trainer.save_model(output_dir=new_path)`. ---
**Author:** ![IdoAmit198](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/idoamit198/32/12110_2.png) [@IdoAmit198](https://discuss.huggingface.co/u/IdoAmit198)\ **Post date:** [December 12, 2022, 7:55am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/17 "2022-12-12T07:55:54Z")
Hey @artificial-cerebral , can you please share a code example of how you do that? ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 20, 2023, 12:57am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/18 "2023-05-20T00:57:56Z")
HI @prachi12 , I hope you are well. sorry dusing training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can sav ethe best model , my code is as follow ```auto training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True, report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path) Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset, eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]), 'attention_mask': torch.stack([f[1] for f in data]), 'labels': torch.stack([f[0] for f in data])}).train() ``` ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 20, 2023, 12:58am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/19 "2023-05-20T00:58:31Z")
Hi @IdoAmit198 , I hope you are well. sorry dusing training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can sav ethe best model , my code is as follow ```auto training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True, report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path) Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset, eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]), 'attention_mask': torch.stack([f[1] for f in data]), 'labels': torch.stack([f[0] for f in data])}).train() ``` ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 20, 2023, 12:59am UTC](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/20 "2023-05-20T00:59:04Z")
HI @Vinayaks117 , I hope you are well. sorry dusing training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can sav ethe best model , my code is as follow ```auto training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True, report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path) Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset, eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]), 'attention_mask': torch.stack([f[1] for f in data]), 'labels': torch.stack([f[0] for f in data])}).train() ``` [Next page](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442.md?page=2)