# Model pre-training precision database: fp16, fp32, bf16 **URL:** \ **Category:** 🤗Transformers\ **Created:** [April 21, 2021, 11:08pm UTC](https://discuss.huggingface.co/t/model-pre-training-precision-database-fp16-fp32-bf16/5671 "2021-04-21T23:08:01Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![stas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stas/32/19289_2.png) [@stas](https://discuss.huggingface.co/u/stas)\ **Post date:** [April 21, 2021, 11:08pm UTC](https://discuss.huggingface.co/t/model-pre-training-precision-database-fp16-fp32-bf16/5671/1 "2021-04-21T23:08:01Z")
There is an emerging need to know how a given model was pre-trained: fp16, fp32, bf16. So one won’t try to use fp32-pretrained model in fp16 regime. And most recently we are bombarded with users attempting to use bf16-pretrained (bfloat16!) models under fp16, which is very problematic since [fp16 and bf16 numerical ranges don’t overlap too well](https://discuss.huggingface.co/t/mixed-precision-for-bfloat16-pretrained-models/5315). We are [discussing adding a new field to 🤗 models](https://github.com/huggingface/transformers/issues/11209) that will tell users how it was trained. Some papers don’t disclose how a model was trained. So perhaps other ways can be found. I made this into a wiki post, so if you can help to compile the knowledge base - and you know for sure which mode it belongs too and you can cite a reference - please add an entry below. Notes: - Let’s focus on official models here - i.e. ones with papers (since some models have hundreds of derived/finetuned checkpoints) - One entry per model is enough unless you know that different checkpoints were trained with different precision which is for example the case with `EleutherAI/gpt-neo` checkpoints (one in bf16 and another in fp32) - Typically if a model was trained on TPU v2 or higher it’s almost sure it’s `bfloat16`. - We are looking for definitive data with references where it clearly states how the model was trained. If you are not sure then please don’t add anything. Thank you! **This is a WIKI post so please add the data directly** # Precision of Pre-trained Models (Wiki) ## float16 (mixed precision) - `allenai/longformer` - [paper](https://arxiv.org/pdf/2004.05150.pdf), “we employed mixed precision training (floating points 16 and 32) using apex12 to reduce memory consumption and speed-up training. However, we kept the attention computation in fp32 to avoid numerical instability issues.” - `allenai/led` - same as `allenai/longformer` - [lvwerra/codeparrot](https://huggingface.co/lvwerra/codeparrot) - [informed by the creator of the model](https://twitter.com/lvwerra/status/1474359681435025413) - `facebook/m2m100_418M` (and others) [train info](https://github.com/pytorch/fairseq/tree/main/examples/m2m_100) - `eleutherai/gpt-neox-20b` (doesn’t exist yet, but including for the sake of future-proofing) - as shown [in the configs](https://mystic.the-eye.eu/public/AI/models/GPT-NeoX-20B/slim_weights/configs/). The [paper](http://eaidata.bmk.sh/data/GPT_NeoX_20B.pdf) also states that the model was in fp16, see “Appendix B: Full Configuration Details.” Finally, Stella Biderman’s [official announcement on Twitter](https://twitter.com/BlancheMinerva/status/1491621129336897536?s=20&t=VAHP-FodMea4kGcAnKmirw) also includes a link to download both “full” weights and “slim” weights which implies mixed precision was used. ## bfloat16 (mixed precision) - `google/mobilebert` - [paper](https://www3.cs.stonybrook.edu/~arunab/papers/acl20.pdf), “we train IB-BERTLARGE on 256 TPU v3 chips” - `eleutherai/gpt-neo-1.3b` - shown in the [config file](https://mystic.the-eye.eu/public/AI/gptneo-release/GPT3_XL/config.json) - `eleutherai/gpt-j-6b` - shown in the [GitHub readme](https://github.com/kingoflolz/mesh-transformer-jax) - `google/pegasus-cnn_dailymail` - XXX: needs reference - `google/pegasus-xsum` - XXX: needs reference - `google/mt5` - most likely same as t5 - `t5` - [paper](https://arxiv.org/pdf/1910.10683) “TPU v3 chips” - `bigscience/T0` and other `T0*` models (trained on TPUs, confirmed on bigscience slack) ## float32 (full precision) - `EleutherAI/gpt-neo-2.7B` - the model’s [config file](https://mystic.the-eye.eu/public/AI/gptneo-release/GPT3_2-7B/config.json) doesn’t specify precision and the [codebase](https://github.com/EleutherAI/gpt-neo/blob/5a2c7d89eba43937eb429158196c51b6b69467c8/model_fns.py#L35) defaults to fp32 - [gsarti/it5-base](https://huggingface.co/gsarti/it5-base) and other `it5-*` - [stated by creator](https://twitter.com/gsarti_/status/1474517412343164935) (JAX-trained) **Please keep your comments on topic, it should be easy to start a new thread if you have related questions/issues to discuss.** ---
**Author:** ![stellaathena](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stellaathena/32/3991_2.png) [@stellaathena](https://discuss.huggingface.co/u/stellaathena)\ **Post date:** [February 11, 2022, 7:25pm UTC](https://discuss.huggingface.co/t/model-pre-training-precision-database-fp16-fp32-bf16/5671/2 "2022-02-11T19:25:00Z")
I’ve updated the listings for EleutherAI models. I decided to include GPT-NeoX 20B in the list for the sake of future-proofing, and so I don’t need to come back and re-document it. T5 is listed as being trained with mixed precision, but I had a conversation with Colin Raffel today that implies that it was trained in fp32. I will follow up and seek documentation. ---
**Author:** ![stellaathena](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stellaathena/32/3991_2.png) [@stellaathena](https://discuss.huggingface.co/u/stellaathena)\ **Post date:** [February 23, 2022, 12:18am UTC](https://discuss.huggingface.co/t/model-pre-training-precision-database-fp16-fp32-bf16/5671/3 "2022-02-23T00:18:14Z")
> [@stellaathena](#): > > T5 is listed as being trained with mixed precision, but I had a conversation with Colin Raffel today that implies that it was trained in fp32. I will follow up and seek documentation. This was a miscommunication - T5 was trained in mixed precision. ---
**Author:** ![stas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stas/32/19289_2.png) [@stas](https://discuss.huggingface.co/u/stas)\ **Post date:** [February 23, 2022, 12:46am UTC](https://discuss.huggingface.co/t/model-pre-training-precision-database-fp16-fp32-bf16/5671/4 "2022-02-23T00:46:06Z")
Thank you for following through, Stella! That’s very helpful! ---
**Author:** ![stas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stas/32/19289_2.png) [@stas](https://discuss.huggingface.co/u/stas)\ **Post date:** [December 3, 2022, 2:18am UTC](https://discuss.huggingface.co/t/model-pre-training-precision-database-fp16-fp32-bf16/5671/7 "2022-12-03T02:18:39Z")
Please keep your comments on topic, it should be easy to start a new thread if you have related questions/issues to discuss. If possible please remove the unrelated comments from here and start new threads, as off-topic only invites more unrelated comments. Thank you