# Quantized Model size difference when using Optimum vs. Onnxruntime

**URL:** <https://discuss.huggingface.co/t/quantized-model-size-difference-when-using-optimum-vs-onnxruntime/20323>\
**Category:** 🤗Optimum\
**Created:** [July 13, 2022, 10:02am UTC](https://discuss.huggingface.co/t/quantized-model-size-difference-when-using-optimum-vs-onnxruntime/20323 "2022-07-13T10:02:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shamik](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@Shamik](https://discuss.huggingface.co/u/Shamik)\
**Post date:** [July 13, 2022, 10:02am UTC](https://discuss.huggingface.co/t/quantized-model-size-difference-when-using-optimum-vs-onnxruntime/20323/1 "2022-07-13T10:02:07Z")

</div>

Package versions  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/6/6a49490e8b32f5a8fffe9462dd0ca58830427e62.png)  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/c/c309775a2583d1009b5d550a6432840274fdecbf.png)  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/f/f010a1fad858e2ea50c4df2c1aec048b9e7053a8.png)  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/b/bf08358bf146e5c1d51dcab6c3beaaf7cce81688.png)

While exporting a question answering model (“deepset/minilm-uncased-squad2”) to ONNX and quantizing it(dynamic quantization) with Optimum, the model size is 68 MB.  
The same model exported while using ONNXRuntime is 32 MB.  
Why is there a difference between both the exported models when the model is the same and the quantization too ?

Optimum Code to convert the model to ONNX and Quantization.

```python
from pathlib import Path
from optimum.onnxruntime import ORTModelForQuestionAnswering, ORTOptimizer
from optimum.onnxruntime.configuration import AutoQuantizationConfig, OptimizationConfig
from optimum.onnxruntime import ORTQuantizer
from optimum.pipelines import pipeline
from transformers import AutoTokenizer

model_checkpoint = "deepset/minilm-uncased-squad2"
save_directory = Path.home()/'onnx/optimum/minilm-uncased-squad2'
save_directory.mkdir(exist_ok=True,parents=True)
file_name = "minilm-uncased-squad2.onnx"
onnx_path = save_directory/"minilm-uncased-squad2.onnx"
# Load a model from transformers and export it through the ONNX format
model = ORTModelForQuestionAnswering.from_pretrained(model_checkpoint, from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
# Save the onnx model and tokenizer
model.save_pretrained(save_directory, file_name=file_name)
tokenizer.save_pretrained(save_directory)

# Define the quantization methodology
qconfig = AutoQuantizationConfig.avx2(is_static=False, per_channel=True)
quantizer = ORTQuantizer.from_pretrained(model_checkpoint, feature="question-answering")
# Apply dynamic quantization on the model
quantizer.export(
    onnx_model_path=onnx_path,
    onnx_quantized_model_output_path= save_directory/"minilm-uncased-squad2-quantized.onnx",
    quantization_config=qconfig,
)
quantizer.model.config.save_pretrained(save_directory)
Path(save_directory/"minilm-uncased-squad2-quantized.onnx").stat().st_size/1024**2

```

ONNX Runtime Code

```python
from transformers.convert_graph_to_onnx import convert
from transformers import AutoTokenizer
from pathlib import Path

model_ckpt = "deepset/minilm-uncased-squad2"
onnx_model_path = Path("../../onnx/minilm-uncased-squad2.onnx")
tokenizer= AutoTokenizer.from_pretrained(model_ckpt)
convert(framework="pt", model=model_ckpt, tokenizer=tokenizer, 
        output=onnx_model_path, opset=12, pipeline_name="question-answering")

from onnxruntime.quantization import quantize_dynamic, QuantType
onnx_model_path = Path("../../../onnx/minilm-uncased-squad2.onnx")
model_output = "../../onnx/minilm-uncased-squad2.quant.onnx"
quantize_dynamic(onnx_model_path, model_output, weight_type=QuantType.QInt8)
Path(model_output).stat().st_size/1024**2

```

Thank you

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [July 14, 2022, 8:09am UTC](https://discuss.huggingface.co/t/quantized-model-size-difference-when-using-optimum-vs-onnxruntime/20323/2 "2022-07-14T08:09:32Z")

</div>

I see you already opened [issue](https://github.com/huggingface/optimum/issues/290) at optimum: It makes sense to focus the discussion on one platform.

Feel free to either close the issue or this thread.

---

<div class="post-metadata">

**Author:** ![Shamik](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@Shamik](https://discuss.huggingface.co/u/Shamik)\
**Post date:** [July 14, 2022, 8:32am UTC](https://discuss.huggingface.co/t/quantized-model-size-difference-when-using-optimum-vs-onnxruntime/20323/3 "2022-07-14T08:32:40Z")

</div>

Absolutely.

Apologies for the duplicate discussion.

---

<div class="post-metadata">

**Author:** ![Shamik](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@Shamik](https://discuss.huggingface.co/u/Shamik)\
**Post date:** [July 14, 2022, 8:33am UTC](https://discuss.huggingface.co/t/quantized-model-size-difference-when-using-optimum-vs-onnxruntime/20323/4 "2022-07-14T08:33:21Z")

</div>

Please follow the discussion here.  
[Quantized Model size difference when using Optimum vs. Onnxruntime · Issue #290 · huggingface/optimum (github.com)](https://github.com/huggingface/optimum/issues/290)