# How to optimize ONNX seq2seq model?

**URL:** <https://discuss.huggingface.co/t/how-to-optimize-onnx-seq2seq-model/20882>\
**Category:** 🤗Optimum\
**Created:** [July 26, 2022, 11:53am UTC](https://discuss.huggingface.co/t/how-to-optimize-onnx-seq2seq-model/20882 "2022-07-26T11:53:16Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pablojs](https://avatars.discourse-cdn.com/v4/letter/p/e19adc/32.png) [@pablojs](https://discuss.huggingface.co/u/pablojs)\
**Post date:** [July 26, 2022, 11:53am UTC](https://discuss.huggingface.co/t/how-to-optimize-onnx-seq2seq-model/20882/1 "2022-07-26T11:53:16Z")

</div>

Hi!

I am trying to improve the performance of `sshleifer/distilbart-cnn-12-6` summarization model using Optimum. Recently tested with other classification models and just by converting to ONNX I was getting an inference performance gain, but seems not to be the case with seq2seq models.

I would like to know the process to optimize this kind of models, because the model is splitted into multiple parts and not supported by the regular optimizer.

You can have a look at this [notebook gist](https://gist.github.com/jspablo/c9ac72ab7d2a89fe58868ef9711af384) where I was trying to optimize each of the ONNX files produced after saving while measuring the performance vs the standard summarization pipeline.

Thank you

---

<div class="post-metadata">

**Author:** ![echarlaix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echarlaix/32/21754_2.png) [@echarlaix](https://discuss.huggingface.co/u/echarlaix)\
**Post date:** [July 27, 2022, 8:12am UTC](https://discuss.huggingface.co/t/how-to-optimize-onnx-seq2seq-model/20882/2 "2022-07-27T08:12:48Z")

</div>

Hi @pablojs,

To optimize a seq2seq model, you should first export it to the ONNX format using `ORTModelForSeq2SeqLM` and then apply optimization on each of its component (`encoder` , `decoder` and `decoder_with_past`). We are currently working on the refactorization of the `ORTOptimizer` in order to simplify its usage, you can follow the progress in [#294](https://github.com/huggingface/optimum/pull/294). You might also be interested in applying dynamic quantization on your model with the `ORTQuantizer` (refactorization in [#270](https://github.com/huggingface/optimum/pull/270))

---

<div class="post-metadata">

**Author:** ![echarlaix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echarlaix/32/21754_2.png) [@echarlaix](https://discuss.huggingface.co/u/echarlaix)\
**Post date:** [August 25, 2022, 8:24am UTC](https://discuss.huggingface.co/t/how-to-optimize-onnx-seq2seq-model/20882/4 "2022-08-25T08:24:17Z")

</div>

Hi @Z3K3, let’s move this discussion to [Optimize an ONNX Seq2Seq model](https://discuss.huggingface.co/t/optimize-an-onnx-seq2seq-model/21807) as you are describing the same problem there. Please don’t cross-post the same question in multiple topics in the future as it makes things difficult to track.