# Difficulty with checkpoint saving and loading (trainer+ FSDP accelerate)

**URL:** <https://discuss.huggingface.co/t/difficulty-with-checkpoint-saving-and-loading-trainer-fsdp-accelerate/79634>\
**Category:** Beginners\
**Created:** [April 1, 2024, 10:08am UTC](https://discuss.huggingface.co/t/difficulty-with-checkpoint-saving-and-loading-trainer-fsdp-accelerate/79634 "2024-04-01T10:08:53Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ayushjain123](https://avatars.discourse-cdn.com/v4/letter/a/c77e96/32.png) [@ayushjain123](https://discuss.huggingface.co/u/ayushjain123)\
**Post date:** [April 1, 2024, 10:08am UTC](https://discuss.huggingface.co/t/difficulty-with-checkpoint-saving-and-loading-trainer-fsdp-accelerate/79634/1 "2024-04-01T10:08:53Z")

</div>

Hi,  
I am fine-tuning a 13B parameter using FSDP in Hugginface Trainer using accelerate. I want to load the best model at the end for which I have set the “load\_best\_model\_at\_end = True”, “save\_total\_limit=1” and “save\_only\_model=True”. The checkpointing is happening as expected, with older checkpoints being deleted automatically. However, while loading the best chekpoint at the end I am getting the error `FileNotFoundError: [Errno 2] No such file or directory: 'output-dir/checkpoint-63/pytorch_model_fsdp.bin`. I checked the checkpoint directory and found the checkpoints are being saved in shards as “pytorch\_model-00001-of-00011.bin”.  
How can I save the checkpoints in unsharded format so that it can be loaded, or enable loading the sharded checkpoint, either of which should resolve this issue.  
@sgugger