# Optimize an ONNX Seq2Seq model

**URL:** <https://discuss.huggingface.co/t/optimize-an-onnx-seq2seq-model/21807>\
**Category:** 🤗Optimum\
**Created:** [August 18, 2022, 8:49pm UTC](https://discuss.huggingface.co/t/optimize-an-onnx-seq2seq-model/21807 "2022-08-18T20:49:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Z3K3](https://avatars.discourse-cdn.com/v4/letter/z/a3d4f5/32.png) [@Z3K3](https://discuss.huggingface.co/u/Z3K3)\
**Post date:** [August 18, 2022, 8:49pm UTC](https://discuss.huggingface.co/t/optimize-an-onnx-seq2seq-model/21807/1 "2022-08-18T20:49:41Z")

</div>

I am trying to optimize a `Seq2Seq` model for summarization, this [guide](https://huggingface.co/docs/optimum/main/en/onnxruntime/quantization#quantize-seq2seq-models) has been very useful for quantizing it, but the results for the quantized model aren’t accurate, so i want to optimize the model and compare the results. I want to know how i can optimize a Seq2seq model, I know that i will have to optimize each the `encoder`, `decoder`, and `decoder_with_past`, but i don’t know how, this is what i have so far:

This is the code i’m trying for optimizing each part, but isn’t working:

```auto
import re
import torch
from transformers import AutoConfig, AutoModelForSeq2SeqLM, AutoTokenizer
import time
from optimum.onnxruntime import ORTQuantizer, ORTModelForSeq2SeqLM, ORTOptimizer
from optimum.onnxruntime.configuration import AutoQuantizationConfig, OptimizationConfig
from pathlib import Path

# load Seq2Seq model and set model file directory
model_id = "facebook/bart-large-cnn"
optimization_config = OptimizationConfig(optimization_level=99) 
onnx_path = '/'

# Create encoder optimizer
encoder_optimizer = ORTOptimizer.from_pretrained(model_name_or_path="/testing/encoder_model.onnx", feature='seq2seq-lm')
encoder_optimizer.export(optimization_config=optimization_config,
                        onnx_model_path=onnx_path / "encoder.onnx",
                        onnx_optimized_model_output_path=onnx_path / "encoder_optimized.onnx")

# Create decoder optimizer
decoder_optimizer = ORTOptimizer.from_pretrained(model_name_or_path="/testing/decoder_model.onnx", feature='seq2seq-lm')
decoder_optimizer.export(optimization_config=optimization_config,
                        onnx_model_path=onnx_path / "decoder.onnx",
                        onnx_optimized_model_output_path=onnx_path / "decoder_optimized.onnx")

# Create decoder with past key values optimizer
decoder_wp_optimizer = ORTOptimizer.from_pretrained(model_name_or_path="/testing/decoder_with_past_model.onnx", feature='seq2seq-lm')
decoder_wp_optimizer.export(optimization_config=optimization_config,
                        onnx_model_path=onnx_path / "decoder_wp.onnx",
                        onnx_optimized_model_output_path=onnx_path / "decoder_wp_optimized.onnx")

OUTPUT:
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.

```

Any help would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![echarlaix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echarlaix/32/21754_2.png) [@echarlaix](https://discuss.huggingface.co/u/echarlaix)\
**Post date:** [August 23, 2022, 4:07pm UTC](https://discuss.huggingface.co/t/optimize-an-onnx-seq2seq-model/21807/2 "2022-08-23T16:07:15Z")

</div>

Hi @Z3K3,

We are currently working on the refactorization of the `ORTOptimizer` in order to simplify its usage, you can follow the progress in [#294](https://github.com/huggingface/optimum/pull/294). You can find an example on how to apply optimization on a Seq2Seq model in the associated [documentation](https://moon-ci-docs.huggingface.co/docs/optimum/pr_294/en/onnxruntime/optimization).

---

<div class="post-metadata">

**Author:** ![mineshj1291](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mineshj1291/32/37149_2.png) [@mineshj1291](https://discuss.huggingface.co/u/mineshj1291)\
**Post date:** [November 17, 2022, 12:09pm UTC](https://discuss.huggingface.co/t/optimize-an-onnx-seq2seq-model/21807/3 "2022-11-17T12:09:56Z")

</div>

> [@echarlaix](#):
>
> associated [documentation](https://moon-ci-docs.huggingface.co/docs/optimum/pr_294/en/onnxruntime/optimization)

Hi, I am trying to export decoder with past model and could not find any good doc. The documentation link is not working for me. can you direct me to updated page for the same. thanks.

---

<div class="post-metadata">

**Author:** ![echarlaix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echarlaix/32/21754_2.png) [@echarlaix](https://discuss.huggingface.co/u/echarlaix)\
**Post date:** [November 17, 2022, 2:33pm UTC](https://discuss.huggingface.co/t/optimize-an-onnx-seq2seq-model/21807/4 "2022-11-17T14:33:48Z")

</div>

Hi @mineshj1291,

Check out our [documentation](https://huggingface.co/docs/optimum/onnxruntime/overview) for more information and examples for the ONNX export of Seq2Seq models as well as their optimization with our `ORTOptimizer`.