# Generate 'continuation' for seq2seq models

**URL:** <https://discuss.huggingface.co/t/generate-continuation-for-seq2seq-models/3822>\
**Category:** Intermediate\
**Created:** [February 22, 2021, 5:37pm UTC](https://discuss.huggingface.co/t/generate-continuation-for-seq2seq-models/3822 "2021-02-22T17:37:51Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![PereLluis13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/perelluis13/32/3714_2.png) [@PereLluis13](https://discuss.huggingface.co/u/PereLluis13)\
**Post date:** [February 22, 2021, 5:37pm UTC](https://discuss.huggingface.co/t/generate-continuation-for-seq2seq-models/3822/1 "2021-02-22T17:37:51Z")

</div>

I am not sure if I missed an obvious way to do this, but I didn’t find any.

Basically the idea is that if we have a seq2seq model, let’s say Bart. Right now, one can input the tokens to the encoder in order to start decoding and generating text using **`model.generate()`**, but there doesn’t seem to be a way to add decoder inputs, that is text which we want the generate function to continue.

Using the example at the documentation:

```
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
# Generate Summary
summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])

```

And this outputs `['My friends']`. Let’s say we want to precondition the generation to start with ‘Friends’ instead of ‘My’, it would be cool to have something like:

```
decoder_inputs = tokenizer(['Friends'], max_length=1024, return_tensors='pt')
# Generate Summary
summary_ids = model.generate(inputs['input_ids'], decoder_inputs = decoder_inputs['inputs_ids'] num_beams=4, max_length=5, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])

```

Which would output a summary starting with ‘Friends’. I am aware that it is possible to do a forward pass with explicit **`decoder_inputs`** but I was wondering if there is a way to do this at generation, to take advantage of beam search and such.

Perhaps with the newly added **`prefix_allowed_tokens_fn`** there is a workaround by having it return the desired starting tokens at the beginning of generation but I was wondering if there is a more straight forward way I missed or is this something that would be interesting to add to the **`generate`** functionality.  
Cheers!

---

<div class="post-metadata">

**Author:** ![PereLluis13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/perelluis13/32/3714_2.png) [@PereLluis13](https://discuss.huggingface.co/u/PereLluis13)\
**Post date:** [February 22, 2021, 8:27pm UTC](https://discuss.huggingface.co/t/generate-continuation-for-seq2seq-models/3822/2 "2021-02-22T20:27:25Z")

</div>

So after looking at the code behind **`generate()`** indeed it already incorporates this functionality if **`decoder_input_ids`** is given as input, in a similar fashion as the **`forward`** function:

```
summary_ids = model.generate(inputs['input_ids'], decoder_input_ids = decoder_inputs['inputs_ids'] num_beams=4, max_length=5, early_stopping=True)

```

So once again I am impressed by the amount of capabilities built into the library. I still think it would be beneficial to clarify this functionality somewhere in the documentation.

I won’t delete the post in case someone comes looking for the same.