# FlashAttention-2's 16 bit requirement

**URL:** <https://discuss.huggingface.co/t/flashattention-2s-16-bit-requirement/67069>\
**Category:** 🤗Optimum\
**Created:** [December 26, 2023, 2:38pm UTC](https://discuss.huggingface.co/t/flashattention-2s-16-bit-requirement/67069 "2023-12-26T14:38:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![peterhung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/peterhung/32/22463_2.png) [@peterhung](https://discuss.huggingface.co/u/peterhung)\
**Post date:** [December 26, 2023, 2:38pm UTC](https://discuss.huggingface.co/t/flashattention-2s-16-bit-requirement/67069/1 "2023-12-26T14:38:58Z")

</div>

On this page on [GPU Inference](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2), FA-2 section, it says:

> FlashAttention-2 can only be used when the model’s dtype is `fp16` or `bf16`. Make sure to cast your model to the appropriate dtype and load them on a supported device before using FlashAttention-2.

But then below that, it then says that it can be used with 4bit quantized model:

> # load in 4bit
> 
> model = AutoModelForCausalLM.from\_pretrained(  
> model\_id,  
> load\_in\_4bit=True,  
> attn\_implementation=“flash\_attention\_2”,  
> )

These 2 statements seem contradictory to me, because 4bit model isn’t stored in fp16 right?

Thanks for your clarifications.

---

<div class="post-metadata">

**Author:** ![regisss](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/regisss/32/22467_2.png) [@regisss](https://discuss.huggingface.co/u/regisss)\
**Post date:** [December 26, 2023, 6:31pm UTC](https://discuss.huggingface.co/t/flashattention-2s-16-bit-requirement/67069/2 "2023-12-26T18:31:05Z")

</div>

Hi @peterhung! Indeed, 4-bit and 8-bit quantization through `bitsandbytes` enables to reduce the memory footprint of the model. However, when the output of a layer is being computed, the weights of this layer are casted to 32-bit or 16-bit precision. Which is why it is compatible with Flash Attention.

Here is a more detailed explanation: [Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA](https://huggingface.co/blog/4bit-transformers-bitsandbytes#advanced-usage)

---

<div class="post-metadata">

**Author:** ![peterhung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/peterhung/32/22463_2.png) [@peterhung](https://discuss.huggingface.co/u/peterhung)\
**Post date:** [December 26, 2023, 6:51pm UTC](https://discuss.huggingface.co/t/flashattention-2s-16-bit-requirement/67069/3 "2023-12-26T18:51:14Z")

</div>

Hi @regisss

Thanks for your explanations - they’re clear.

And thanks for the link - I’d check it out as well.