# When exporting seq2seq models with ONNX, why do we need both decoder\_with\_past\_model.onnx and decoder\_model.onnx?

**URL:** <https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354>\
**Category:** 🤗Optimum\
**Created:** [March 8, 2023, 4:23pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354 "2023-03-08T16:23:24Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Xenova](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xenova/32/28089_2.png) [@Xenova](https://discuss.huggingface.co/u/Xenova)\
**Post date:** [March 8, 2023, 4:23pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/1 "2023-03-08T16:23:24Z")

</div>

Hi there, I’m the creator of [Transformers.js](https://github.com/xenova/transformers.js), a JavaScript library which aims to run HuggingFace models directly in the browser. It relies on optimum to convert PyTorch models to ONNX, which can then be used inside web browsers using [onnxruntime-web](https://www.npmjs.com/package/onnxruntime-web). For the most part, everything is working fine, but there appears to be a ton of duplicate parameters between decoder\_with\_past\_model.onnx and decoder\_model.onnx, for sequence to sequence and causal language models.

For causal language models, such as GPT2, I was able to avoid having to use decoder\_model.onnx by creating an empty tensor for past\_key\_values, and passing the inputs through the decoder\_model\_with\_past.onnx (see [here](https://github.com/xenova/transformers.js/blob/c2e47f61396749a77d767d2572e440fd3c5db8b5/src/models.js#L432-L441)).

I tried employing the same trick for sequence to sequence models (e.g., T5, Bart and Whisper), but ran into a problem since they need past key values for the encoder too. Once again, I tried just creating dummy inputs with a dimension of 0 where concatenation would take place. However, this didn’t work either, as the outputted “present” key value pairs were all empty tensors too.

I asked on the Discord channel, and I was directed to the forum to see if anyone could provide some assistance. Perhaps being able to export smaller parts of the models to avoid significant parameter duplication? This would dramatically improve performance (cutting down model size by up to 50%)!

I look forward to any responses! Thanks 🤗

~ Xenova

---

<div class="post-metadata">

**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\
**Post date:** [March 8, 2023, 8:31pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/2 "2023-03-08T20:31:18Z")

</div>

Hi @Xenova , thank you for having a try at this!

I have the same experience as you for gpt2 (although I had to add `position_ids` as inputs to have matching logits due to [this logic](https://github.com/huggingface/transformers/blob/bcc8d30affba29c594320fc80e4a4422fb850175/src/transformers/models/gpt2/modeling_gpt2.py#L795-L802)) using only `decoder_with_past_model.onnx`.

Unfortunately I did not have time to try with encoder-decoder models, I was assuming it was possible. I could have a look shortly.

Alternatively, are you able to support ONNX models that have subgraphs? That’s the approach we are currently taking in Optimum, for reference: [Validating ONNX model fails for GPT-J · Issue #607 · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/issues/607#issuecomment-1432837730) . This is currently available only for decoder-only models for now though, I plan to extend to encoder-decoder architectures.

If you export a decoder-only model you’ll see as output a merged decoder in charge of both cases without past/with past: `optimum-cli export onnx gpt2 gpt2_onnx/`

---

<div class="post-metadata">

**Author:** ![Xenova](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xenova/32/28089_2.png) [@Xenova](https://discuss.huggingface.co/u/Xenova)\
**Post date:** [March 8, 2023, 8:41pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/3 "2023-03-08T20:41:41Z")

</div>

> I have the same experience as you for gpt2 (although I had to add `position_ids` as inputs to have matching logits due to [this logic](https://github.com/huggingface/transformers/blob/bcc8d30affba29c594320fc80e4a4422fb850175/src/transformers/models/gpt2/modeling_gpt2.py#L795-L802)) using only `decoder_with_past_model.onnx` .

Are you still having the problem, even if you pass an empty tensor (of shape [batch\_size, this.num\_heads, 0, this.dim\_kv])? I was able to bypass that check since, in that case, it isn’t None, and then the concatenation logic works.

As you can see, here is the code to create the model (only uses decoder\_with\_past\_model.onnx):

> <https://github.com/xenova/transformers.js/blob/c2e47f61396749a77d767d2572e440fd3c5db8b5/src/models.js#L145-L171>

and it works correctly (see the [demo](https://xenova.github.io/transformers.js/) for proof).

* * *

> Unfortunately I did not have time to try with encoder-decoder models, I was assuming it was possible. I could have a look shortly.

That would be greatly appreciated! I’ve been banging my head against the wall for 2 days now, but I am getting strange outputs (sometimes the logits are NaN, other times the generated present\_key\_values are empty tensors).

> Alternatively, are you able to support ONNX models that have subgraphs? That’s the approach we are currently taking in Optimum, for reference

I should say that I am not an expert in ONNX haha; so, I’m not too sure what subgraphs are 👀 For me, at least, I treat ONNX as a magical black box, and I provide the inputs and process the outputs.

> If you export a decoder-only model you’ll see as output a merged decoder in charge of both cases without past/with past

Okay great! Yes, this is something I desperately need for Transformers.js … the smaller the model is, the better!

---

<div class="post-metadata">

**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\
**Post date:** [March 8, 2023, 8:54pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/4 "2023-03-08T20:54:48Z")

</div>

Oh, I see, so that `past_key_values[0][0].size(-2)` is zero. I did not have a try at it - did not expect ONNX Runtime to work with zero-sized inputs, but I’ll have a try!

Subgraphs is basically a fancy way to handle controlflows (if/else, loops) with ONNX. Depending on whether you need to use k/v cache or not, you can dispatch the compute on one or the other of the two branches of an `If` node, that shares weights between both:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/7ddd06ffa989bf31574b7986e5c17b64b866ba46.png)

As a rule of thumb though, if you are able to avoid subgraphs (e.g. thanks to the hack with fake k/v), I would recommend to avoid it.

---

<div class="post-metadata">

**Author:** ![Xenova](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xenova/32/28089_2.png) [@Xenova](https://discuss.huggingface.co/u/Xenova)\
**Post date:** [March 8, 2023, 8:58pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/5 "2023-03-08T20:58:38Z")

</div>

> [@fxmarty](#):
>
> Oh, I see, so that `past_key_values[0][0].size(-2)` is zero. I did not have a try at it - did not expect ONNX Runtime to work with zero-sized inputs, but I’ll have a try!

Haha yes, I was quite surprised myself when I got it working with gpt2.

> As a rule of thumb though, if you are able to avoid subgraphs (e.g. thanks to the hack with fake k/v), I would recommend to avoid it.

Understood 👍 . Is there anything I can do to help try get this working? I could continue trying myself to get the empty tensor working for encoder-decoder models. (Although, I do think fixing [this issue](https://github.com/huggingface/optimum/issues/869) will either fix it, or help get closer to our goal! 🙂 )

---

<div class="post-metadata">

**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\
**Post date:** [March 8, 2023, 9:11pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/6 "2023-03-08T21:11:35Z")

</div>

If you think there are inputs/outputs that are ill-defined, you could have a look at [optimum/optimum/exporters/onnx at main · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/tree/main/optimum/exporters/onnx) (in order of hierarchy: `base.py`, `config.py`, `model_configs.py` as explained in [here](https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/contribute), and hack and try from there, for example [optimum/base.py at 913f9d5fe619acc852210f5978c62e1271185287 · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/blob/913f9d5fe619acc852210f5978c62e1271185287/optimum/exporters/onnx/base.py#L747-L774) and [optimum/config.py at 913f9d5fe619acc852210f5978c62e1271185287 · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/blob/913f9d5fe619acc852210f5978c62e1271185287/optimum/exporters/onnx/config.py#L173-L199).

I’ll have a look if there’s an issue with the export itself (like whisper it seems), or if it’s fully an implementation problem.

---

<div class="post-metadata">

**Author:** ![Xenova](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xenova/32/28089_2.png) [@Xenova](https://discuss.huggingface.co/u/Xenova)\
**Post date:** [March 9, 2023, 1:15pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/7 "2023-03-09T13:15:08Z")

</div>

HUGE update! I just got whisper seq2seq working WITHOUT having to load both decoders. This is done using the merged decoder.

There were some hacks still necessary though, namely:

1. Creating empty dummy past key values for encoder and decoder when using cache\_branch

```auto
let encoder_heads = this.config.encoder_attention_heads;
let encoder_dims = [1, encoder_heads, 0, this.config.d_model / encoder_heads];
for (let i = 0; i < this.config.encoder_layers; ++i) {
    decoderFeeds[`past_key_values.${i}.encoder.key`] = new Tensor('float32', [], encoder_dims)
    decoderFeeds[`past_key_values.${i}.encoder.value`] = new Tensor('float32', [], encoder_dims)
}

let decoder_heads = this.config.decoder_attention_heads;
let decoder_dims = [1, decoder_heads, 0, this.config.d_model / decoder_heads];
for (let i = 0; i < this.config.decoder_layers; ++i) {
    decoderFeeds[`past_key_values.${i}.decoder.key`] = new Tensor('float32', [], decoder_dims)
    decoderFeeds[`past_key_values.${i}.decoder.value`] = new Tensor('float32', [], decoder_dims)
}

```

1. I don’t know if onnx supports simple booleans, so I just created a tensor with a single boolean: `use_cache_branch: new Tensor('bool', [pastKeyValues !== null], [1])`. Perhaps the if logic checks for all values to be true, and so, it works. In other words, I’m not questioning it too much haha

2. There were also a ton of warnings produces, but, it works… so, I don’t really care 😉 :  

* * *

I will see if it is as simple to convert the other models types. 🤞

---

<div class="post-metadata">

**Author:** ![Hannan](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@Hannan](https://discuss.huggingface.co/u/Hannan)\
**Post date:** [March 10, 2023, 5:07pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/8 "2023-03-10T17:07:32Z")

</div>

I’m facing the exact problem with whisper model. Could you please specify where you have applied your hacking codes in the conversion?

---

<div class="post-metadata">

**Author:** ![Xenova](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xenova/32/28089_2.png) [@Xenova](https://discuss.huggingface.co/u/Xenova)\
**Post date:** [March 12, 2023, 3:37am UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/9 "2023-03-12T03:37:32Z")

</div>

Here’s a link to the source code: [transformers.js/models.js at main · xenova/transformers.js · GitHub](https://github.com/xenova/transformers.js/blob/main/src/models.js)

Hopefully that will be enough for you to decipher! 😄

---

<div class="post-metadata">

**Author:** ![AhmedMEGZ](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@AhmedMEGZ](https://discuss.huggingface.co/u/AhmedMEGZ)\
**Post date:** [June 20, 2023, 4:10pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/10 "2023-06-20T16:10:32Z")

</div>

@Xenova we are trying to make an App using Quantized whisper-finetuned model and run it in the browser using react our problem that we don’t know how to generate predicted output text from last\_hidden\_state and logits

> **[GitHub - iMegz/onnxruntime-web-whisper](https://github.com/iMegz/onnxruntime-web-whisper)**
>
> Contribute to iMegz/onnxruntime-web-whisper development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![regisss](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/regisss/32/22467_2.png) [@regisss](https://discuss.huggingface.co/u/regisss)\
**Post date:** [January 10, 2024, 2:48pm UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/13 "2024-01-10T14:48:38Z")

</div>

Hi @Noahloghman! It seems it couldn’t find the model. What is the model you were trying to load?

---

<div class="post-metadata">

**Author:** ![Noahloghman](https://avatars.discourse-cdn.com/v4/letter/n/dbc845/32.png) [@Noahloghman](https://discuss.huggingface.co/u/Noahloghman)\
**Post date:** [March 7, 2024, 5:15am UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/14 "2024-03-07T05:15:11Z")

</div>

Hi @regisss sorry for the late reply, its Llama 27B from hugging face.  
Thank you

---

<div class="post-metadata">

**Author:** ![Noahloghman](https://avatars.discourse-cdn.com/v4/letter/n/dbc845/32.png) [@Noahloghman](https://discuss.huggingface.co/u/Noahloghman)\
**Post date:** [March 7, 2024, 5:19am UTC](https://discuss.huggingface.co/t/when-exporting-seq2seq-models-with-onnx-why-do-we-need-both-decoder-with-past-model-onnx-and-decoder-model-onnx/33354/15 "2024-03-07T05:19:30Z")

</div>

Hi guys Hi @xenova I have this issue :  
[E:onnxruntime:, inference\_session.cc:1533 onnxruntime::InferenceSession::Initialize::\<lambda\_9a5ue43270b854edk3er320c0a5c4y9a\>::operator ()] Exception during initialization: D:\a\_work\1\s\onnxruntime\core\optimizer\initializer.cc:31 onnxruntime::Initializer::Initializer !model\_path.IsEmpty() was false. model\_path must not be empty. Ensure that a path is provided when the model  
is created or loaded.

I think this still an issue