# Control EncoderDecoderModel to generate tokens step by step

**URL:** <https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756>\
**Category:** 🤗Transformers\
**Created:** [October 26, 2020, 4:10pm UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756 "2020-10-26T16:10:45Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 26, 2020, 4:10pm UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/1 "2020-10-26T16:10:45Z")

</div>

Hey, I am writing a model and the baseline is bert2bert for text summarization. But I want to add a specific layer above the Decoder. For example , I want to change the LMhead of Decoder by concatenating another vector. But the DecoderModel outputs all the hidden states at once. I want to control it for step by step decoding. In other words. I want to use the concatenated vector as the hidden state for generation and use the generated word vector for next step’s input. How can I change the model or call the interface properly ?

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 26, 2020, 6:00pm UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/2 "2020-10-26T18:00:31Z")

</div>

My expression may not be very clear. I want to say, in EncoderDeocderModel, I load the model like this

```auto
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')

```

I want to modify the structure of LMHead or manipulate the single step of the output hidden state of the decoder to make the use-specific generation . Is it possible ?

---

<div class="post-metadata">

**Author:** ![claartje-barkhof](https://avatars.discourse-cdn.com/v4/letter/c/71c47a/32.png) [@claartje-barkhof](https://discuss.huggingface.co/u/claartje-barkhof)\
**Post date:** [October 27, 2020, 9:20am UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/4 "2020-10-27T09:20:26Z")

</div>

You can just run the model multiple times to generate a conditional sequence. You for example first give the start token as input\_ids, make a prediction, add that token to the input\_sequence and forward again. This will give you a pure conditionally generated sequence. It is however not very efficient as it does not make use of ‘cache’, i.e. it will compute the outputs for every time step, also if it has already outputted values for that time step in the previous generation step. You could fix that by adding a notion of cache to the code (returning previous key, values, which you can later add). GPT-2 implements this cache with variables names past and present.

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 27, 2020, 9:38am UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/5 "2020-10-27T09:38:17Z")

</div>

@claartje-barkhof ,thank you for your response . Can you provide a example for bert2bert model ?  
because I build the model like this

```auto
  model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')

```

And when I run the forward method like this

```auto
outputs = model(input_ids=src, decoder_input_ids=dst, labels=dst, return_dict=True)

```

It outputs all the logits at once.  
I don’t konw how to control it .

---

<div class="post-metadata">

**Author:** ![claartje-barkhof](https://avatars.discourse-cdn.com/v4/letter/c/71c47a/32.png) [@claartje-barkhof](https://discuss.huggingface.co/u/claartje-barkhof)\
**Post date:** [October 27, 2020, 9:47am UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/6 "2020-10-27T09:47:06Z")

</div>

If you want to generate incrementally it should be something along the lines of. Read this as pseudocode 🙂

```auto
generated_so_far = [start_token]
while eos_token not in generated_so_far:
    outputs = model(..., decoder_inputs_ids=generated_so_far, ...) # incrementally build up the decoder_inputs with your previous predictions. This way your prediction becomes dependent on both the encoder inputs and the previous outputs
    next_token_logits = outputs[0][0, -1, :] # this indexing depends on the model, but take the last hidden state of the last token
    filtered_logits = top_k_top_p_filtering(next_token_logits, top_k=top_k, top_p=top_p)
    next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=1) 
    generated_so_far = torch.cat((generated_so_far, next_token.unsqueeze(0)), dim=1)

```

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 27, 2020, 11:23am UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/7 "2020-10-27T11:23:52Z")

</div>

Thank you very much !  
I want to ask another question .  
Do you know how much the speed will be reduced using this step by step method if the ‘cache’ mechanism is not implemented ?  
It seems that every single generation step the whole source text will pass the encoder model . However, It only needs to pass once.

---

<div class="post-metadata">

**Author:** ![claartje-barkhof](https://avatars.discourse-cdn.com/v4/letter/c/71c47a/32.png) [@claartje-barkhof](https://discuss.huggingface.co/u/claartje-barkhof)\
**Post date:** [October 27, 2020, 1:50pm UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/8 "2020-10-27T13:50:15Z")

</div>

That very much depends on the size of the model, batch size, the sequence length you want/expect to generate and on your available resources.

I was trying to do without cache with a GPT2 the other day and had issues quite soon. Longer sequences took way too long: I guess already with batch size 8, sequence length 30, GPT2-base model it started to get really slow on my local machine. It depends also if you want to decode a lot of sequences at once etc. Just for trying a few example, you might be okay without the cache.

I would recommend reading this blog [about cache](http://jalammar.github.io/illustrated-gpt2/) to understand the concept and then check the code of GPT-2 to see how it is implemented.

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 27, 2020, 4:48pm UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/9 "2020-10-27T16:48:24Z")

</div>

Thank you !

---

<div class="post-metadata">

**Author:** ![dmlls](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dmlls/32/19929_2.png) [@dmlls](https://discuss.huggingface.co/u/dmlls)\
**Post date:** [June 8, 2022, 2:49pm UTC](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756/11 "2022-06-08T14:49:48Z")

</div>

@claartje-barkhof @guoziyuan Hi guys, almost two years later but this thread came in very handy!

Could you guys expand a bit more on how to perform caching? I read the Illustrated GPT-2 blog, but it would be great to have a more concrete example on how to achieve it. For reference, I’m using a BART model for summarization.