# Shouldn't \`\_flash\_attn\_2\_enabled\` be documented?

**URL:** <https://discuss.huggingface.co/t/shouldnt-flash-attn-2-enabled-be-documented/64070>\
**Category:** 🤗Transformers\
**Created:** [November 30, 2023, 3:46am UTC](https://discuss.huggingface.co/t/shouldnt-flash-attn-2-enabled-be-documented/64070 "2023-11-30T03:46:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dinhanhx](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dinhanhx/32/20115_2.png) [@dinhanhx](https://discuss.huggingface.co/u/dinhanhx)\
**Post date:** [November 30, 2023, 3:46am UTC](https://discuss.huggingface.co/t/shouldnt-flash-attn-2-enabled-be-documented/64070/1 "2023-11-30T03:46:12Z")

</div>

While reading the Llama code, I found out that we can use flash attention via option `flash_attn_2_enabled` at [these lines](https://github.com/huggingface/transformers/blob/3eceaa3637197fa78dd3525cb3df57fcaf5ba00d/src/transformers/models/llama/modeling_llama.py#L597C37-L598). However, this can not be seen in [LlamaConfig](https://huggingface.co/docs/transformers/v4.35.2/model_doc/llama#transformers.LlamaConfig).

After bit googling, I think to use flash attention we need [Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention) right?

---

<div class="post-metadata">

**Author:** ![marcsun13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcsun13/32/19059_2.png) [@marcsun13](https://discuss.huggingface.co/u/marcsun13)\
**Post date:** [November 30, 2023, 5:24pm UTC](https://discuss.huggingface.co/t/shouldnt-flash-attn-2-enabled-be-documented/64070/2 "2023-11-30T17:24:24Z")

</div>

Hi @dinhanhx, we integrated flashattention2 recently. You can learn more about the integration [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one). [Note](https://github.com/huggingface/transformers/issues/26350) that not every model supports flashattention2.