# Support for LLaMA in EncoderDecoder framework

**URL:** <https://discuss.huggingface.co/t/support-for-llama-in-encoderdecoder-framework/35908>\
**Category:** 🤗Transformers\
**Created:** [April 7, 2023, 4:23pm UTC](https://discuss.huggingface.co/t/support-for-llama-in-encoderdecoder-framework/35908 "2023-04-07T16:23:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Varal7](https://avatars.discourse-cdn.com/v4/letter/v/3ec8ea/32.png) [@Varal7](https://discuss.huggingface.co/u/Varal7)\
**Post date:** [April 7, 2023, 4:23pm UTC](https://discuss.huggingface.co/t/support-for-llama-in-encoderdecoder-framework/35908/1 "2023-04-07T16:23:55Z")

</div>

I’m trying to use LLaMA as a drop-in replacement for GPT2 in my ViT-GPT2 model.

After seeing issue [Using FNet model in Encoder Decoder Models · Issue #22308 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/22308), it seems like HuggingFace doesn’t plan to support future models in the EncoderDecoder framework and I should adapt the model to suit my own needs.

I’m planning to follow the steps described in [Trying to add support for GPT2 as decoder in EncoderDecoder model · Issue #4483 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/4483)

Are there any gotchas I should know about?

---

<div class="post-metadata">

**Author:** ![JORGEDC01](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jorgedc01/32/42834_2.png) [@JORGEDC01](https://discuss.huggingface.co/u/JORGEDC01)\
**Post date:** [March 8, 2025, 9:08pm UTC](https://discuss.huggingface.co/t/support-for-llama-in-encoderdecoder-framework/35908/2 "2025-03-08T21:08:51Z")

</div>

Hey, Did you try to do it? I’m trying to merge a bge-small-1.5 encoder with a Llama2-1B with EncoderDecoderModel. The issue is that Llama2 is decoder-only and doesn’t support cross attention, so the EncoderDecoderModel don’t let me join both