# Enabling Flash Attention 2

**URL:** <https://discuss.huggingface.co/t/enabling-flash-attention-2/87652>\
**Category:** 🤗Transformers\
**Created:** [May 21, 2024, 12:50pm UTC](https://discuss.huggingface.co/t/enabling-flash-attention-2/87652 "2024-05-21T12:50:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\
**Post date:** [May 21, 2024, 12:50pm UTC](https://discuss.huggingface.co/t/enabling-flash-attention-2/87652/1 "2024-05-21T12:50:04Z")

</div>

What is the difference between using Flash Attention 2 via

`model = AutoModelForCausalLM.from_pretrained(ckpt, attn_implementation = "sdpa")`

vs

`model = AutoModelForCausalLM.from_pretrained(ckpt, attn_implementation = "flash_attention_2")`

when Pytorch SDPA support FA2 according to [docs](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.htm) ?

@marcsun13

---

<div class="post-metadata">

**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\
**Post date:** [May 29, 2024, 9:54am UTC](https://discuss.huggingface.co/t/enabling-flash-attention-2/87652/2 "2024-05-29T09:54:33Z")

</div>

@ybelkada Can you shed some light on this?

---

<div class="post-metadata">

**Author:** ![saireddy](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@saireddy](https://discuss.huggingface.co/u/saireddy)\
**Post date:** [July 3, 2024, 2:41pm UTC](https://discuss.huggingface.co/t/enabling-flash-attention-2/87652/3 "2024-07-03T14:41:43Z")

</div>

@varadhbhatnagar were you able to figure out the difference?