# Mixed precision for bfloat16-pretrained models **URL:** \ **Category:** 🤗Transformers\ **Created:** [April 5, 2021, 8:06pm UTC](https://discuss.huggingface.co/t/mixed-precision-for-bfloat16-pretrained-models/5315 "2021-04-05T20:06:20Z")\ **Posts on this page:** 1\ **Showing post:** 3
**Author:** ![stas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stas/32/19289_2.png) [@stas](https://discuss.huggingface.co/u/stas)\ **Post date:** [April 21, 2021, 11:37pm UTC](https://discuss.huggingface.co/t/mixed-precision-for-bfloat16-pretrained-models/5315/3 "2021-04-21T23:37:27Z")
We started compiling a wiki of how different models were pre-trained, please add your knowledge there - thanks! > [@Model pre-training precision database: fp16, fp32, bf16](https://discuss.huggingface.co/t/compiling-data-on-how-models-were-pre-trained-fp16-fp32-bf16/5671): > > There is an emerging need to know how a given model was pre-trained: fp16, fp32, bf16. So one won’t try to use fp32-pretrained model in fp16 regime. And most recently we are bombarded with users attempting to use bf16-pretrained (bfloat16!) models under fp16, which is very problematic since [fp16 and bf16 numerical ranges don’t overlap too well](https://discuss.huggingface.co/t/mixed-precision-for-bfloat16-pretrained-models/5315). We are [discussing adding a new field to hugs models](https://github.com/huggingface/transformers/issues/11209) that will tell users how it was trained. Some papers don’t disclose how a model was trained. So … --- _[View the full topic](https://discuss.huggingface.co/t/mixed-precision-for-bfloat16-pretrained-models/5315)._