# Caching encoder state for multiple encoder-decoder \`.generate()\` calls?

**URL:** <https://discuss.huggingface.co/t/caching-encoder-state-for-multiple-encoder-decoder-generate-calls/81126>\
**Category:** 🤗Transformers\
**Created:** [April 10, 2024, 7:01pm UTC](https://discuss.huggingface.co/t/caching-encoder-state-for-multiple-encoder-decoder-generate-calls/81126 "2024-04-10T19:01:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbarrow](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jbarrow/32/25675_2.png) [@jbarrow](https://discuss.huggingface.co/u/jbarrow)\
**Post date:** [April 10, 2024, 7:01pm UTC](https://discuss.huggingface.co/t/caching-encoder-state-for-multiple-encoder-decoder-generate-calls/81126/1 "2024-04-10T19:01:33Z")

</div>

I’m using a `VisionEncoderDecoderModel` and I want to reuse the encoded image to decode multiple times (say, 30+ times per image). However, I don’t want to rerun the encoder every time I call `model.generate()`. Is there a way to cache the encoder state and reuse it? Or is there another efficient way to decode multiple times from the encoded input?

---

<div class="post-metadata">

**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\
**Post date:** [April 12, 2024, 7:41am UTC](https://discuss.huggingface.co/t/caching-encoder-state-for-multiple-encoder-decoder-generate-calls/81126/2 "2024-04-12T07:41:30Z")

</div>

Hi! You can pass into “generate” an argument called “encoder\_outputs” which will be used by the decoder then, instead of running encoder every time. Optionally you can pass in “decoder\_input\_ids”, otherwise it will be initialized from BOS token.

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("facebook/bart-base")
model = AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-base")

inputs_encoder = tok("Hello, my dog is cute", return_tensors="pt")
decoder_input_ids = tok("Bonjour", return_tensors="pt")["input_ids"]

encoder_outputs = model.get_encoder()(**inputs_encoder)
out = model.generate(decoder_input_ids=decoder_input_ids, encoder_outputs=encoder_outputs, num_beams=1, do_sample=False)
print(tok.batch_decode(out))

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [April 12, 2024, 7:41pm UTC](https://discuss.huggingface.co/t/caching-encoder-state-for-multiple-encoder-decoder-generate-calls/81126/3 "2024-04-12T19:41:46Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.