# How to generate a sequence using inputs\_embeds instead of input\_ids?

**URL:** <https://discuss.huggingface.co/t/how-to-generate-a-sequence-using-inputs-embeds-instead-of-input-ids/4145>\
**Category:** 🤗Transformers\
**Created:** [March 4, 2021, 9:56am UTC](https://discuss.huggingface.co/t/how-to-generate-a-sequence-using-inputs-embeds-instead-of-input-ids/4145 "2021-03-04T09:56:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Euna](https://avatars.discourse-cdn.com/v4/letter/e/8dc957/32.png) [@Euna](https://discuss.huggingface.co/u/Euna)\
**Post date:** [March 4, 2021, 9:56am UTC](https://discuss.huggingface.co/t/how-to-generate-a-sequence-using-inputs-embeds-instead-of-input-ids/4145/1 "2021-03-04T09:56:19Z")

</div>

Hello, I am struggling with generating a sequence of tokens using model.generate() with inputs\_embeds.  
For my research, I have to use inputs\_embeds (word embedding vectors) instead of input\_ids (token indices) as an input to the GPT2 model.  
I want to employ model.generate() which is a convenient tool for generating a sequence of tokens, but there is no argument for inputs\_embeds. I tried to edit " transformers.generation\_utils", but it was not easy to figure out which lines I should change.

Is there any idea that I can easily generate tokens with default settings for hyper-paremeters as in model.generate()? If there is any idea, help me please.

---

<div class="post-metadata">

**Author:** ![StackSmasher](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@StackSmasher](https://discuss.huggingface.co/u/StackSmasher)\
**Post date:** [October 18, 2021, 2:01pm UTC](https://discuss.huggingface.co/t/how-to-generate-a-sequence-using-inputs-embeds-instead-of-input-ids/4145/2 "2021-10-18T14:01:22Z")

</div>

Were you able to figure something out for this ? My needs are similar. I’m using BART model

---

<div class="post-metadata">

**Author:** ![sadakmed](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sadakmed/32/2908_2.png) [@sadakmed](https://discuss.huggingface.co/u/sadakmed)\
**Post date:** [October 18, 2021, 10:07pm UTC](https://discuss.huggingface.co/t/how-to-generate-a-sequence-using-inputs-embeds-instead-of-input-ids/4145/3 "2021-10-18T22:07:22Z")

</div>

@StackSmasher,

consider you have the tensor `inputs_embeds` which I believe will be in the shape of `(batch_size, seq_length, dim)`, or If you have a `hidden_state` in the shape of `(batch_size, dim)` just `unsqueeze(dim=1)` it to become `(batch_size,1,dim)`

Then smoothly u can achieve the desired output by:

- create an `attention mask` `(batch_size, seq_len)`
- create `decoder_input_ids` `(batch_size, 1)`

tracing these two parameters, leads to two functions below:

attention mask you are already familiar with it.

> <https://github.com/huggingface/transformers/blob/d5ff69fce92bb1aab9273d674e762a8eddcb2e3f/src/transformers/generation_utils.py#L394-L403>

`decoder_input_ids` it’s `ones` too (multiplied by config.decoder\_start\_token\_ids).

> <https://github.com/huggingface/transformers/blob/d5ff69fce92bb1aab9273d674e762a8eddcb2e3f/src/transformers/generation_utils.py#L419-L426>

wrapping it all:

```auto
generator = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
inputs_embeds = # a 3D tensor, [batch, seq_length, dim]
attention_mask = torch.ones(inputs_embeds.shape[:2], dtype=torch.long)
decoder_input_ids = torch.ones((inputs_embeds.shape[0], 1), dtype=torch.long)*generator.config.decoder_start_token_id
output_ids = generator.generate(attention_mask=attention_mask, decoder_input_ids=decoder_input_ids, inputs_embeds=inputs_embeds, max_length=100,num_beams=4)

```

---

<div class="post-metadata">

**Author:** ![StackSmasher](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@StackSmasher](https://discuss.huggingface.co/u/StackSmasher)\
**Post date:** [November 25, 2021, 10:47am UTC](https://discuss.huggingface.co/t/how-to-generate-a-sequence-using-inputs-embeds-instead-of-input-ids/4145/4 "2021-11-25T10:47:14Z")

</div>

Hey thanks a lot for the reply and it works nice !  
Slight clarification, would I need to set `decoder_input_ids` when I’m training the said model using `inputs_embeds` as well ?

---

<div class="post-metadata">

**Author:** ![anon3699016](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@anon3699016](https://discuss.huggingface.co/u/anon3699016)\
**Post date:** [April 17, 2022, 4:59pm UTC](https://discuss.huggingface.co/t/how-to-generate-a-sequence-using-inputs-embeds-instead-of-input-ids/4145/5 "2022-04-17T16:59:06Z")

</div>

For BART, you can use encoder\_outputs, which you should get from the encoder part of BART model.