# Adding cross-attention to custom models

**URL:** <https://discuss.huggingface.co/t/adding-cross-attention-to-custom-models/5408>\
**Category:** 🤗Transformers\
**Created:** [April 10, 2021, 1:06pm UTC](https://discuss.huggingface.co/t/adding-cross-attention-to-custom-models/5408 "2021-04-10T13:06:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sachin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachin/32/26534_2.png) [@sachin](https://discuss.huggingface.co/u/sachin)\
**Post date:** [April 10, 2021, 1:06pm UTC](https://discuss.huggingface.co/t/adding-cross-attention-to-custom-models/5408/1 "2021-04-10T13:06:07Z")

</div>

Hi, I was thinking of adding cross attention between a visual transformer and a bert model. Was wondering if there was a way that I could do this using the HF library.

What I was thinking was if somewhere in the HF Bert model API if I had access to where it took in the queries, keys, and values, I could subclass the BERT submodule and add cross attention instead of just having self attention. I’m visualizing something very much like [this code snippet from the annotated transformer paper](https://nlp.seas.harvard.edu/2018/04/03/attention.html#decoder). Specifically the `DecodeLayer` class where there is `self_attn` as well an additional `src_attn` which I would need to add in.

I am also aware that I would need to copy the weights for everything but the `src_attn` module. Just need some mechanism to do so. Fingers crossed there is some place in the HF API that I can do exactly that.

Happy to do this myself if someone can point where in the HF library I should be looking at to see where it uses queries, keys and values arguments.

---

<div class="post-metadata">

**Author:** ![sachin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachin/32/26534_2.png) [@sachin](https://discuss.huggingface.co/u/sachin)\
**Post date:** [September 4, 2021, 12:04pm UTC](https://discuss.huggingface.co/t/adding-cross-attention-to-custom-models/5408/2 "2021-09-04T12:04:05Z")

</div>

bump. Sorry guys just wondering if anyone had any ideas about this.

---

<div class="post-metadata">

**Author:** ![sachin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachin/32/26534_2.png) [@sachin](https://discuss.huggingface.co/u/sachin)\
**Post date:** [October 21, 2022, 11:56am UTC](https://discuss.huggingface.co/t/adding-cross-attention-to-custom-models/5408/3 "2022-10-21T11:56:14Z")

</div>

Partial answer:

```auto
model1 = AutoModel.from_pretrained("gpt2")
gpt_config = model1.config
gpt_config.add_cross_attention = True
new_model = AutoModelForCausalLM.from_pretrained("gpt2", config=gpt_config)

```

Similarly for models like bert you need to do one additional step like this:

```auto
model1 = AutoModel.from_pretrained("bert-base-cased")
bert_config = model1.config
bert_config.add_cross_attention = True
bert_config.is_decoder = True
model2 = AutoModel.from_pretrained("bert-base-cased", config=bert_config)

```