# Checkpoints and disk storage **URL:** \ **Category:** šŸ¤—Transformers\ **Created:** [October 7, 2020, 8:34pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412 "2020-10-07T20:34:27Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![shainaraza](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shainaraza/32/2854_2.png) [@shainaraza](https://discuss.huggingface.co/u/shainaraza)\ **Post date:** [October 7, 2020, 8:34pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/1 "2020-10-07T20:34:27Z")
I am fine tuning custom dataset using this link [https://huggingface.co/transformers/master/custom\_datasets.html](https://huggingface.co/transformers/master/custom_datasets.html) it is saving too many checkpoints that make my disk full, any suggestions? thanks ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [October 8, 2020, 1:03pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/2 "2020-10-08T13:03:53Z")
If itā€™s the part using `Trainer`, you can use the argument `save_total_limit` (in `TrainingArguments`) to limit the number of checkpoints kept. ---
**Author:** ![shainaraza](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shainaraza/32/2854_2.png) [@shainaraza](https://discuss.huggingface.co/u/shainaraza)\ **Post date:** [October 8, 2020, 1:19pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/3 "2020-10-08T13:19:18Z")
thanks, indeed its very useful for me, One more question for my clarification, once it is saved, should I use the last checkpoint for using the model later? or my concept is not correct? Also I want to know how many checkpoints should we have? can we make it one checkpoint? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [October 8, 2020, 1:20pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/4 "2020-10-08T13:20:27Z")
There is a `load_best_model` argument too, that will automatically load your best model (according to a metric you choose). Itā€™s all in the [docs](https://huggingface.co/transformers/master/main_classes/trainer.html#transformers.TrainingArguments) šŸ˜‰ ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [October 8, 2020, 1:20pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/5 "2020-10-08T13:20:39Z")
@sgugger How ā€œsmartā€ is this feature? I remember that in openNMT when you specify a max checkpoint this does not take into account the best evaluated checkpoint up to that point. In other words, if there is an old checkpoint that is the best, it can still get deleted. That should not be the case, so I am wondering whether the implementation in the transformers trainer is a bit smarter? ---
**Author:** ![shainaraza](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shainaraza/32/2854_2.png) [@shainaraza](https://discuss.huggingface.co/u/shainaraza)\ **Post date:** [October 8, 2020, 1:21pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/6 "2020-10-08T13:21:23Z")
thanks for information, I am looking into the doc ---
**Author:** ![shainaraza](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shainaraza/32/2854_2.png) [@shainaraza](https://discuss.huggingface.co/u/shainaraza)\ **Post date:** [October 8, 2020, 2:26pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/7 "2020-10-08T14:26:25Z")
thanks , it solved my issue, esp working with ā€™ **load\_best\_model\_at\_end** ā€™ ---
**Author:** ![shainaraza](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shainaraza/32/2854_2.png) [@shainaraza](https://discuss.huggingface.co/u/shainaraza)\ **Post date:** [October 8, 2020, 2:48pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/8 "2020-10-08T14:48:29Z")
However for this **load\_best\_model\_at\_end** , does it mean the models saved during checkpoints, for example, I am using distillBert, the best model here refers to the checkpoint? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [October 9, 2020, 1:06pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/9 "2020-10-09T13:06:10Z")
It is smarter: the best checkpoint is always put at the top of the list of checkpoints, so it never gets deleted (or if it does, itā€™s a bug šŸ˜‰ ). ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [October 9, 2020, 1:53pm UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/10 "2020-10-09T13:53:54Z")
Love it. Thanks! ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 20, 2023, 1:14am UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/11 "2023-05-20T01:14:37Z")
hi @BramVanroy, I hope you are well. sorry during training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell me how I can save the best model , my code is as follow, what I missed in the code? ```auto training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True, report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path) Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset, eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]), 'attention_mask': torch.stack([f[1] for f in data]), 'labels': torch.stack([f[0] for f in data])}).train() ``` ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [May 23, 2023, 11:19am UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/12 "2023-05-23T11:19:55Z")
> [@SUNM](#): > > `save_total_limit=1` `save_total_limit=1` will always only keep one checkpoint. Remove this argument and try again. ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 26, 2023, 1:10am UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/13 "2023-05-26T01:10:29Z")
@BramVanroy many thanks for your reply. Sorry, I need to get training and validation loss to make graph. do you know how I can get the value from train\_state.log\_history and save them? I really appreciate your big help. ---
**Author:** ![mirix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mirix/32/16327_2.png) [@mirix](https://discuss.huggingface.co/u/mirix)\ **Post date:** [May 26, 2023, 7:22am UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/14 "2023-05-26T07:22:12Z")
I use: save\_total\_limit=1 evaluation\_strategy=ā€˜stepsā€™ load\_best\_model\_at\_end=True In my very limited experience, as I started in this business a couple of days ago, I believe that save\_total\_limit=n keeps n+1 models as the best model is always kept and always first. So, if you set it to 1, at any given time you will have the working model and the best model. ---
**Author:** ![EswarAleti](https://avatars.discourse-cdn.com/v4/letter/e/ec9cab/32.png) [@EswarAleti](https://discuss.huggingface.co/u/EswarAleti)\ **Post date:** [July 24, 2023, 9:23am UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/15 "2023-07-24T09:23:46Z")
@sgugger @BramVanroy transformers.TrainingArguments( per\_device\_train\_batch\_size=8, gradient\_accumulation\_steps=16, warmup\_steps=100, num\_train\_epochs=2, learning\_rate=2e-5, fp16=True, logging\_steps=1, output\_dir=ā€œlora-alpacaā€, save\_total\_limit=3, ) Iā€™m able to generate checkpoint-500 directory with following structure i. optimizer.pt ii. pytorch\_model.bin iii. rng\_state.pth iv. scaler.pt v. scheduler.pt vi. trainer\_state.json vii. training\_args.json **But the real problem starts here** When I ran the same code in different notebook file, I canā€™t see any checkpoint-500 folder Whatā€™s the issue and resolution? Please help me out! ---
**Author:** ![Aravindan](https://avatars.discourse-cdn.com/v4/letter/a/cab0a1/32.png) [@Aravindan](https://discuss.huggingface.co/u/Aravindan)\ **Post date:** [June 2, 2024, 11:51am UTC](https://discuss.huggingface.co/t/checkpoints-and-disk-storage/1412/16 "2024-06-02T11:51:47Z")
is it possible to save the outputs in some other volume like ```python training_args = TrainingArguments(output_dir=" ``` Is it possible?