# T5 fp16 issue is fixed

**URL:** <https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139>\
**Category:** 🤗Transformers\
**Created:** [January 12, 2021, 12:08pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139 "2021-01-12T12:08:33Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [January 12, 2021, 12:08pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/1 "2021-01-12T12:08:33Z")

</div>

We have just fixed the T5 fp16 issue for some of the T5 models!

(Announcing it here, since lots of users were facing this issue and T5 is one most widely used model in the library)

**TL;DR:**

Previously, there was an issue when using `T5` models in `fp16`; it was producing `nan` `loss` and `logits`. Now on the master, this issue is fixed for the following T5 models and versions. Now you should be able to train and use these models for inference in `fp16` and see a decent speed-up!

- **T5v1** : `t5-small` , `t5-base` , `t5-large`
- **T5v1\_1** : `google/t5-v1_1-small` , `google/t5-v1_1-base`
- **MT5** : `google/mt5-small` , `google/mt5-base`

For those of you who are interested, here’s a description of what was causing `nan` loss and how it is fixed.

`t5-small` was the only T5 model that was working in fp16. The rest of the models produce `nan` loss/logits.

for all the models and versions (v1, v1.1, mT5), at some point, we get `inf` values in `hidden_states` after applying the final linear layer (`wo`) in `T5DenseReluDense` and `T5DenseGatedGeluDense`.

> <https://github.com/huggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L248-L278>

which results in `nan` values in `T5LayerNorm`.

Also for `t5-large`, `t5-v1_1-base`, `t5-v1_1-large`, there are `inf` values in the output of `T5LayerSelfAttention` and `T5LayerCrossAttention`, specifically where we add the attn output with the `hidden_states`

> <https://github.com/huggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L548>

> <https://github.com/huggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L584>

This happens during both training and inference, to reproduce

## fix

To avoid `inf` values we could clamp the `hidden_states` to the max values for the current data type if there are `inf` in it. i.e

```python
if torch.isinf(hidden_states).any():
    clamp_value = torch.finfo(hidden_states.dtype).max - 1000
    hidden_states = torch.clamp(hidden_states, min=-clamp_value, max=clamp_value)

```

we need to add this after self attn, cross-attn, and the feed-forward layer which is where the `inf` values occur. This works for both `apex` and `amp`

To verify this fix, trained `t5-base`, `t5-v1_1-base` and `t5-v1_1-small` on `cnn/dm` for 10k steps (1.11 epochs)  
Here’s the training command, to run this navigate to `examples/seq2seq` dir, follow the instructions in the readme to download `cnn_dm` and dataset, and then run the following command

```bash
export M=google/t5-v1_1-base
export OUT_DIR=t5-v1_1-base-cnn-fp16
export DATA_DIR=cnn_dm

python finetune_trainer.py \
    --model_name_or_path $M \
    --data_dir $DATA_DIR \
    --output_dir $OUT_DIR --overwrite_output_dir \
    --max_steps=10000 \
    --gradient_accumulation_steps=8 \
    --learning_rate=1e-4 \
    --per_device_train_batch_size=4 \
    --n_val 500 \
    --max_target_length=56 --val_max_target_length=128 \
    --fp16 --fp16_backend apex \
    --do_train --do_eval --evaluation_strategy steps \
    --logging_steps=100 --logging_first_step --eval_steps=2500 --save_steps=2500 --save_total_limit=2 \
    --sortish_sampler \

```

for evaluation

```bash
python run_eval.py \
    t5-v1_1-base-cnn-fp16 cnn_dm/test.source hypothesis.txt \
    --reference_path cnn_dm/test.target \
    --score_path metrics.json \
    --device cuda:0 \
    --prefix summarize: \
    --bs 16 \
    --fp16 \

```

and got the following metrics (ROUGE2)

1. for `t5-base`: 19.2804
2. for `t5-v1.1-base`: 18.4316  
(note that the score for `t5-base` is more because it’s already pre-trained on `cnn/dm`)

To compare this, evaluated the pre-trained `t5-base` in both `fp32` and `fp16`, which gave the following results

1. `fp16`: 18.3681
2. `fp32`: 18.394

So the results are close enough.

To verify the fix for `t5-large`, evaluated the pre-trained `t5-large` in `fp32` and `fp16` (use the same command above to evaluate `t5-large`) and got the following results

1. `fp16`: 19.2734
2. `fp32`: 19.2342

Surprisingly, rouge2 is slightly better in `fp16`.

So with the above fix, the following model types now work in `fp16` (opt level `01`), and give descent speed-up 🙂

- **T5v1** : `t5-small`, `t5-base`, `t5-large`
- **T5v1\_1** : `google/t5-v1_1-small`, `google/t5-v1_1-base`
- **MT5** : `google/mt5-small`, `google/mt5-base`

One interesting observation,  
For inference, the `t5-base` fine-tuned with `fp16` and evaluated in `fp32` is faster (~1.31x) than pre-trained `t5-base` evaluated in `fp16`. See this [colab](https://colab.research.google.com/drive/1UaMBsWp3e1Qf-fYKxXmtulsvPXViKa72?usp=sharing)

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [January 12, 2021, 10:17pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/2 "2021-01-12T22:17:11Z")

</div>

Nice fix!  
The speed discrepancy might be because of different length generations.

---

<div class="post-metadata">

**Author:** ![aaronchavez](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aaronchavez/32/958_2.png) [@aaronchavez](https://discuss.huggingface.co/u/aaronchavez)\
**Post date:** [January 26, 2021, 4:44pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/3 "2021-01-26T16:44:17Z")

</div>

Very cool! Works well on T4 as well.

Any guesses why the inference time for fp16 doesn’t seem to be noticeably faster? Saw only a small difference in the shared colab, and seeing similar behavior locally…

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [January 29, 2021, 9:00am UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/4 "2021-01-29T09:00:02Z")

</div>

Hey @aaronchavez

as explained in this issue [Improve PyTorch examples for FP16 · Issue #9752 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/9752)  
To get the full speed-up of FP16 training, every tensor passed through the model should have all its dimensions be a multiple of 8.

---

<div class="post-metadata">

**Author:** ![aaronchavez](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aaronchavez/32/958_2.png) [@aaronchavez](https://discuss.huggingface.co/u/aaronchavez)\
**Post date:** [January 29, 2021, 3:21pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/5 "2021-01-29T15:21:18Z")

</div>

Thanks for replying! I was wondering about inference in particular (via generate())…is it the same situation, there? I am seeing approximately the same times for generation with fp16 and fp32.

---

<div class="post-metadata">

**Author:** ![Dara](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@Dara](https://discuss.huggingface.co/u/Dara)\
**Post date:** [March 21, 2021, 2:21pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/6 "2021-03-21T14:21:33Z")

</div>

Hi

This is not fixed as far as I see, could you have a look here please:

> <https://github.com/huggingface/transformers/issues/10830>
>
> \## Environment info
> 
> \- \`transformers\` version: 4.5.0.dev0
> \- Platform: Linux-4….15.0-65-generic-x86\_64-with-glibc2.10
> \- Python version: 3.8.8
> \- PyTorch version (GPU?): 1.7.1+cu101 (True)
> \- Tensorflow version (GPU?): not installed (NA)
> \- Using GPU in script?: Yes
> \- Using distributed or parallel set-up in script?: No
> 
> \### Who can help
> @patil-suraj @patrickvonplaten 
> 
> \## Information
> 
> Model I am using (Bert, XLNet ...): t5-large
> 
> The problem arises when using:
> \* \[\] my own modified scripts: run\_seq2seq with minor modifications (attached)
> 
> 
> The tasks I am working on is:
> \* \[\] my own task or dataset: Closed-Book Open Domain QA
> 
> \## To reproduce
> 
> Steps to reproduce the behavior (the fix I'm suggesting is very simple, so perhaps there is no reason to reproduce):
> 1. unzip the attached zip (below).
> 2. run 
> \`\`\`bash
> python run\_seq2seq.py --model\_name\_or\_path=t5-large
> \--do\_train
> \--do\_eval
> \--task=qa
> \--train\_file=data/PAQ.filtered.regular.16000.json
> \--validation\_file=data/PAQ.filtered.regular.16000.json
> \--output\_dir=results/5e-5-t5-large-4096000-128-140-1792000-0.1-regular-true-4
> \--overwrite\_output\_dir
> \--per\_device\_train\_batch\_size=1
> \--per\_device\_eval\_batch\_size=128
> \--predict\_with\_generate
> \--fp16
> \--max\_steps=1000
> \--evaluation\_strategy=steps
> \--text\_column=question
> \--summary\_column=answer
> \--save\_total\_limit=5
> \--cache\_dir=../.cache
> \--save\_steps=500000
> \--learning\_rate=5e-5
> \--eval\_steps=96000
> \--warmup\_steps=100
> \--run\_name=5e-5-t5-large-4096000-128-140-1792000-0.1-regular-true-4
> \--dropout\_rate=0.1
> \--gradient\_accumulation\_steps=1
> \--logging\_steps=1
> \`\`\`
> 
> \## Expected behavior
> Training without nans. 
> 
> \## Possible fix
> I debugged and saw that we get nans at the \`modeling\_t5.py\` script in line 241 
> \`\`\`python
> hidden\_states = hidden\_states \* torch.rsqrt(variance + self.variance\_epsilon)
> \`\`\`
> By modifing this line to:
> \`\`\`python
> clamp\_value = torch.finfo(hidden\_states.dtype).max - 1000
> hidden\_states = torch.clamp(hidden\_states, min=-clamp\_value, max=clamp\_value) \* torch.rsqrt(variance + self.variance\_epsilon)
> \`\`\`
> It seems to be solved.
> 
> BTW it happens in the last layers (this might explain why it wasn't caught in \[this fix\](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139))
> 
> \[seq2seq.zip\](https://github.com/huggingface/transformers/files/6177063/seq2seq.zip)

> <https://github.com/huggingface/transformers/issues/10819>
>
> \## Environment info
> \<!-- You can run the command \`transformers-cli env\` and cop…y-and-paste its output below.
> Don't forget to fill out the missing fields in that output! --\>
> 
> \- \`transformers\` version: 4.4.2
> \- Platform: linux
> \- Python version: 3.7
> \- PyTorch version (GPU?): 1.8 
> \- Tensorflow version (GPU?): - 
> \- Using GPU in script?: - 
> \- Using distributed or parallel set-up in script?: - 
> 
> \### Who can help
> \<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
> If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of \*\*who to tag\*\*.
> Please tag fewer than 3 people.
> 
> Models:
> 
> \- albert, bert, xlm: @LysandreJik
> \- blenderbot, bart, marian, pegasus, encoderdecoder, t5: @patrickvonplaten, @patil-suraj
> \- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
> \- fsmt: @stas00
> \- funnel: @sgugger
> \- gpt2: @patrickvonplaten, @LysandreJik
> \- rag: @patrickvonplaten, @lhoestq
> \- tensorflow: @LysandreJik
> 
> Library:
> 
> \- benchmarks: @patrickvonplaten
> \- deepspeed: @stas00
> \- ray/raytune: @richardliaw, @amogkam
> \- text generation: @patrickvonplaten
> \- tokenizers: @LysandreJik
> \- trainer: @sgugger
> \- pipelines: @LysandreJik
> 
> Documentation: @sgugger
> 
> Model hub:
> 
> \- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.
> 
> HF projects:
> 
> \- nlp datasets: \[different repo\](https://github.com/huggingface/nlp)
> \- rust tokenizers: \[different repo\](https://github.com/huggingface/tokenizers)
> 
> Examples:
> 
> \- maintained examples (not research project or legacy): @sgugger, @patil-suraj
> \- research\_projects/bert-loses-patience: @JetRunner
> \- research\_projects/distillation: @VictorSanh
> 
> --\>
> 
> t5: @patrickvonplaten, @patil-suraj
> 
> \## Information
> 
> I am using mt5-small model:
> \* the problem arises when using fp16 with mt5
> 
> The tasks I am working on is:
> \* translation
> 
> \## To reproduce
> 
> Steps to reproduce the behavior:
> 
> \`python run\_translation.py --model\_name\_or\_path google/mt5-small --do\_train --do\_eval --source\_lang en --target\_lang ro --dataset\_name wmt16 --dataset\_config\_name ro-en --output\_dir test/tst-translation --per\_device\_train\_batch\_size=4 --per\_device\_eval\_batch\_size=4 --overwrite\_output\_dir --predict\_with\_generate --max\_train\_samples 100 --fp16\`
> 
> outputs:
> \`\`\`
> \*\*\*\*\* eval metrics \*\*\*\*\*
> epoch = 3.0
> eval\_bleu = 0.0039
> eval\_gen\_len = 2.95
> eval\_loss = nan
> eval\_mem\_cpu\_alloc\_delta = 4MB
> eval\_mem\_cpu\_peaked\_delta = 5MB
> eval\_mem\_gpu\_alloc\_delta = 0MB
> eval\_mem\_gpu\_peaked\_delta = 1080MB
> eval\_runtime = 72.1865
> eval\_samples = 1999
> eval\_samples\_per\_second = 27.692
> \`\`\`
> 
> 
> \## Expected behavior
> 
> being able to use fp16 with mt5 models. Thank you very much for your help, this is really crucial for me to be able to run these models with fp16 to be able to fit more data into old GPUs I have access to and I appreciate a lot your help.

I really need these models and I appreciate having a look into it.  
thanks a lot

---

<div class="post-metadata">

**Author:** ![Dara](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@Dara](https://discuss.huggingface.co/u/Dara)\
**Post date:** [March 21, 2021, 2:22pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/7 "2021-03-21T14:22:29Z")

</div>

@valhalla @patrickvonplaten

---

<div class="post-metadata">

**Author:** ![FL33TW00D](https://avatars.discourse-cdn.com/v4/letter/f/e0b2c6/32.png) [@FL33TW00D](https://discuss.huggingface.co/u/FL33TW00D)\
**Post date:** [March 21, 2021, 3:58pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/8 "2021-03-21T15:58:10Z")

</div>

Hi Dara,  
Can you specify which model you’re trying to use?

---

<div class="post-metadata">

**Author:** ![Dara](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@Dara](https://discuss.huggingface.co/u/Dara)\
**Post date:** [March 21, 2021, 5:45pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/9 "2021-03-21T17:45:06Z")

</div>

Hi there  
I use mt5-small, more info provided in the link to the issues me and another users opened thanks

---

<div class="post-metadata">

**Author:** ![Savindu](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@Savindu](https://discuss.huggingface.co/u/Savindu)\
**Post date:** [July 24, 2021, 5:11am UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/10 "2021-07-24T05:11:18Z")

</div>

@valhalla I tried “finetune\_trainer.py” you mentioned command.

I got this error.

```auto
07/23/2021 06:26:38 - INFO - __main__ - ***Train***
/usr/local/lib/python3.7/dist-packages/transformers/trainer.py:1026: FutureWarning: `model_path` is deprecated and will be removed in a future version. Use `resume_from_checkpoint` instead.
  FutureWarning,
Traceback (most recent call last):
  File "/content/gdrive/MyDrive/Colab Notebooks/generation/transformers/examples/legacy/seq2seq/finetune_trainer.py", line 367, in <module>
    main()
  File "/content/gdrive/MyDrive/Colab Notebooks/generation/transformers/examples/legacy/seq2seq/finetune_trainer.py", line 305, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1138, in train
    self.create_optimizer_and_scheduler(num_training_steps=max_steps)
  File "/content/gdrive/MyDrive/Colab Notebooks/generation/transformers/examples/legacy/seq2seq/seq2seq_trainer.py", line 118, in create_optimizer_and_scheduler
    if self.sharded_dpp:
AttributeError: 'Seq2SeqTrainer' object has no attribute 'sharded_dpp'

```

I have no idea how to fix this issue. How can I fix this?

---

<div class="post-metadata">

**Author:** ![DarkDragon](https://avatars.discourse-cdn.com/v4/letter/d/ecae2f/32.png) [@DarkDragon](https://discuss.huggingface.co/u/DarkDragon)\
**Post date:** [January 20, 2022, 7:29am UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/11 "2022-01-20T07:29:15Z")

</div>

Hi @valhalla

The fix seems not to be applied on the MT5 model:

1. Nan losses occur when fine-tuning the model with the latest (stable) version of transformers.
2. MT5 works well with fp16.

Do you see the same behavior?

---

<div class="post-metadata">

**Author:** ![Pedrambbk](https://avatars.discourse-cdn.com/v4/letter/p/c89c15/32.png) [@Pedrambbk](https://discuss.huggingface.co/u/Pedrambbk)\
**Post date:** [February 1, 2023, 5:22pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/12 "2023-02-01T17:22:06Z")

</div>

I had the same issue and I changed it to fp16=False and it worked. Although the loss became a non-nan value, it caused the Out Of Memory error frequently. Impossible to train more. I tested many things and finally I tried this:  
fp16\_full\_eval=True instead of fp16l=True. It solved the loss problem but at the end ROUGE metrics became all zero. How can it compute the loss but not the metrics?

---

<div class="post-metadata">

**Author:** ![Pedrambbk](https://avatars.discourse-cdn.com/v4/letter/p/c89c15/32.png) [@Pedrambbk](https://discuss.huggingface.co/u/Pedrambbk)\
**Post date:** [February 3, 2023, 11:36am UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/13 "2023-02-03T11:36:53Z")

</div>

Are the changes already merged?  
I am unable to fine-tune mT5 with fp16.

---

<div class="post-metadata">

**Author:** ![zokica](https://avatars.discourse-cdn.com/v4/letter/z/b2d939/32.png) [@zokica](https://discuss.huggingface.co/u/zokica)\
**Post date:** [April 5, 2023, 6:38pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/14 "2023-04-05T18:38:13Z")

</div>

@ [valhalla](https://discuss.huggingface.co/u/valhalla)

So where is this fix now? I do not get where I should place code? Is it in anew version of transformers?

---

<div class="post-metadata">

**Author:** ![RajSingh333](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@RajSingh333](https://discuss.huggingface.co/u/RajSingh333)\
**Post date:** [April 27, 2023, 10:37pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/15 "2023-04-27T22:37:35Z")

</div>

I am getting same issue with `google/flan-t5-small` model with `fp16=True`. Is the error reported for this model too?

---

<div class="post-metadata">

**Author:** ![RubenTao0914](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@RubenTao0914](https://discuss.huggingface.co/u/RubenTao0914)\
**Post date:** [July 1, 2023, 2:19pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/16 "2023-07-01T14:19:20Z")

</div>

I am getting the same issue with `Salesforce/blip2-flan-t5-xl` which uses `google/flan-t5-xl` with `amp`.  
I am using `transformers==4.29.2`.  
I got `inf` values in `hidden_states ` after applying the final linear layer (`wo` ) in `T5DenseGatedActDense`.  
I found this by using PyCharm debug mode, as shown in the screenshot below:

 ![屏幕截图 2023-07-01 215553](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/d/edd4e17fe506af4e2b02b8b32963df0b8057afc0.png)  
By the way, I didn’t find the fixing code which clamps the `hidden_states` to the max values described in **fix** in the `T5DenseGatedActDense` class.

---

<div class="post-metadata">

**Author:** ![RubenTao0914](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@RubenTao0914](https://discuss.huggingface.co/u/RubenTao0914)\
**Post date:** [July 1, 2023, 3:14pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/17 "2023-07-01T15:14:54Z")

</div>

I am using `transformers==4.29.2`.  
I may have found a logical flaw in `T5Block` as shown in below:

> Blockquote

# Apply Feed Forward layer

hidden\_states = self.layer-1

# clamp inf values to enable fp16 training

if hidden\_states.dtype == torch.float16:  
clamp\_value = torch.where(  
torch.isinf(hidden\_states).any(),  
torch.finfo(hidden\_states.dtype).max - 1000,  
torch.finfo(hidden\_states.dtype).max,  
)  
hidden\_states = torch.clamp(hidden\_states, min=-clamp\_value, max=clamp\_value)

> Blockquote  
> Apparently the fixing code which clamps the `hidden_states` to the max values only works when `hidden_states` is fp16. It may works fine when the whole model is fp16. But I am trying to do mixed precision training using `torch.amp`, so the `hidden_states` returned by `T5LayerFF` is fp32 because the last operation in `T5LayerFF` is the adding op, a.k.a the residual connection, which can’t autocast to `float16` according to [https://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float16](https://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float16).  
> The screenshot below also proved this theory:
> 
> ![屏幕截图 2023-07-01 225003](https://us1.discourse-cdn.com/hellohellohello/original/3X/f/0/f00629d4900807f918e2810305e5c27a9eb31393.png)

---

<div class="post-metadata">

**Author:** ![lexandstuff](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lexandstuff/32/4018_2.png) [@lexandstuff](https://discuss.huggingface.co/u/lexandstuff)\
**Post date:** [May 27, 2024, 5:00am UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/18 "2024-05-27T05:00:53Z")

</div>

3 year later, it seems t5-large still doesn’t support fp16. Anyone found a solution?

---

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](https://discuss.huggingface.co/u/dblakely)\
**Post date:** [June 20, 2024, 7:10pm UTC](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/19 "2024-06-20T19:10:10Z")

</div>

> [@lexandstuff](#):
>
> 3 year later, it seems t5-large still doesn’t support fp16. Anyone found a solution?

You should use bf16 instead of fp16 for T5 models