# Saving bf16 Model Weights When Using Accelerate+DeepSpeed

**URL:** <https://discuss.huggingface.co/t/saving-bf16-model-weights-when-using-accelerate-deepspeed/98824>\
**Category:** 🤗Accelerate\
**Created:** [July 22, 2024, 9:34pm UTC](https://discuss.huggingface.co/t/saving-bf16-model-weights-when-using-accelerate-deepspeed/98824 "2024-07-22T21:34:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![OxxoCodes](https://avatars.discourse-cdn.com/v4/letter/o/46a35a/32.png) [@OxxoCodes](https://discuss.huggingface.co/u/OxxoCodes)\
**Post date:** [July 22, 2024, 9:34pm UTC](https://discuss.huggingface.co/t/saving-bf16-model-weights-when-using-accelerate-deepspeed/98824/1 "2024-07-22T21:34:32Z")

</div>

Hello! What’s the correct way to save a model that’s being trained using Zero3 and bf16 mixed precision via the Accelerate implementation of DeepSpeed?

The closest I can find is setting the `stage3_gather_16bit_weights_on_model_save` flag in the config, but from my understanding setting this to true would save it as `fp16` , while leaving it as false would require me to run `zero_to_fp32.py`, which would save the model weights as fp32 .

How do I preserve the bf16 weights?

---

<div class="post-metadata">

**Author:** ![jpcorb20](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jpcorb20](https://discuss.huggingface.co/u/jpcorb20)\
**Post date:** [November 11, 2024, 2:05pm UTC](https://discuss.huggingface.co/t/saving-bf16-model-weights-when-using-accelerate-deepspeed/98824/2 "2024-11-11T14:05:18Z")

</div>

Hello, same question here!

---

<div class="post-metadata">

**Author:** ![jpcorb20](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jpcorb20](https://discuss.huggingface.co/u/jpcorb20)\
**Post date:** [March 17, 2025, 3:16pm UTC](https://discuss.huggingface.co/t/saving-bf16-model-weights-when-using-accelerate-deepspeed/98824/3 "2025-03-17T15:16:42Z")

</div>

any update on this?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 17, 2025, 5:02pm UTC](https://discuss.huggingface.co/t/saving-bf16-model-weights-when-using-accelerate-deepspeed/98824/4 "2025-03-17T17:02:58Z")

</div>

Hmm… by [Hugging Chat](https://huggingface.co/chat/):

* * *

To save a model trained using ZeRO-3 and BF16 mixed precision via the Accelerate implementation of DeepSpeed, you need to ensure that the model weights are saved in their partitioned form while preserving the BF16 precision. Here is how you can achieve this:

* * *

### Key Points:

1. **ZeRO-3 and BF16 Configuration** :

2. **Saving the Model** :

3. **Model Saving Process** :

4. **Loading the Saved Model** :

* * *

### Example Configuration:

Here’s an example of how to configure the DeepSpeed settings for your use case:

```python
# DeepSpeed config file (.json)
{
    "bf16": {"enabled": true},
    "zero_optimization": {
        "stage": 3,
        "stage3_gather_16bit_weights_on_model_save": false,
        "offload_optimizer": {"device": "none"},
        "offload_param": {"device": "none"}
    }
}

```

This configuration ensures that:

- The model is trained using BF16 mixed precision.
- The weights remain in BF16 format during training.
- The weights are saved in their partitioned BF16 form when saving the model.

* * *

### Final Answer:

To save a model trained using Zero3 and BF16 mixed precision while preserving the BF16 weights:

1. **Ensure the following configuration in your DeepSpeed settings** :

2. **Save the model normally** :

By following these steps, you will successfully save the model weights in their native BF16 format for future use without unnecessary conversions [1][2].

---

<div class="post-metadata">

**Author:** ![jpcorb20](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jpcorb20](https://discuss.huggingface.co/u/jpcorb20)\
**Post date:** [March 17, 2025, 10:01pm UTC](https://discuss.huggingface.co/t/saving-bf16-model-weights-when-using-accelerate-deepspeed/98824/5 "2025-03-17T22:01:34Z")

</div>

Works quite well indeed. Thanks a lot for the clear explanations ! 🙂

On my end, I had to use `trainer.save_model(output_directory)` to save the model checkpoint correctly without errors. Then, I could run the `zero_to_fp32.py` to get the `pytorch_model.bin`, and finally I can load with `.from_pretrained(...)` and convert to safetensors to recover the same BF16 output model as other configurations (e.g. zero stage 1 or 2). Thanks so much again!