# Does fp16 training compromise accuracy?

**URL:** <https://discuss.huggingface.co/t/does-fp16-training-compromise-accuracy/18014>\
**Category:** Models\
**Created:** [May 16, 2022, 3:37pm UTC](https://discuss.huggingface.co/t/does-fp16-training-compromise-accuracy/18014 "2022-05-16T15:37:37Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![nbroad](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbroad/32/26465_2.png) [@nbroad](https://discuss.huggingface.co/u/nbroad)\
**Post date:** [May 17, 2022, 9:49pm UTC](https://discuss.huggingface.co/t/does-fp16-training-compromise-accuracy/18014/3 "2022-05-17T21:49:45Z")

</div>

Mixed precision training (fp16) is only possible on certain hardware and in some cases results in [training instability](https://discuss.huggingface.co/t/mixed-precision-for-bfloat16-pretrained-models/5315) depending on if the model was pre-trained using bfloat16.

For older GPUs (before Volta/Turing), fp16 provides no speed up and will require more memory because both the fp16 values and fp32 values will be stored in memory. I’d recommend reading this: [Performance and Scalability: How To Fit a Bigger Model and Train It Faster](https://huggingface.co/docs/transformers/performance#fp16-training)

---

_[View the full topic](https://discuss.huggingface.co/t/does-fp16-training-compromise-accuracy/18014)._