# Supporting ONNX optimized models **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 27, 2020, 4:06pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891 "2020-08-27T16:06:29Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![bdalal](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@bdalal](https://discuss.huggingface.co/u/bdalal)\ **Post date:** [August 27, 2020, 4:06pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/1 "2020-08-27T16:06:29Z")
In a lot of cases, ONNX optimized models offer much better performance benefits as compared to using PyTorch models. This performance boost coupled with the pipelines offered by HuggingFace are a really great combo for delivering a great experience both in terms of inference speed and model performance. Right now, it’s possible to use ONNX models with a little bit of modification to the pipeline.py code. On my tests on the QuestionAnsweringPipeline with SQuADv2 dataset, I see performance improvements of 1.5-2x with models like [`bert-large-uncased-whole-word-masking-finetuned-squad`](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad) and [`distilbert-base-cased-distilled-squad`](https://huggingface.co/distilbert-base-cased-distilled-squad) I’m wondering if this is something that the devs/community considers worthwhile for supporting directly in the transformers repo. I’m doing some work on this for my own project, but it’s mostly a hacky impl. at this point. If there’s greater interest in this then I could try to integrate ONNX support for inference more fully in the code base. Would love to hear everyone’s thoughts. Thanks ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [August 31, 2020, 1:57pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/2 "2020-08-31T13:57:15Z")
I know @mfuntowicz is working on this so he may have more input. ---
**Author:** ![a-ware](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@a-ware](https://discuss.huggingface.co/u/a-ware)\ **Post date:** [September 1, 2020, 7:24pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/3 "2020-09-01T19:24:59Z")
We would love to see onnx support in the pipelines, it would make serving models much more scalable. Actually we have never worked with onnx, but the statistics look pretty good. What do you think about automatically generating onnx models next to TF and pytorch models, so over the time any uploaded model on HF hub would be available as onnx model too. We would love to help you there, but need to read more into onnx runtime. ---
**Author:** ![a-ware](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@a-ware](https://discuss.huggingface.co/u/a-ware)\ **Post date:** [September 6, 2020, 5:45am UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/4 "2020-09-06T05:45:49Z")
> Very cool pipelines implementation ---
**Author:** ![dataiku](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dataiku/32/2634_2.png) [@dataiku](https://discuss.huggingface.co/u/dataiku)\ **Post date:** [October 14, 2020, 8:03am UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/5 "2020-10-14T08:03:37Z")
Hi there, Are there any plans for Huggingface to distribute pre-trained models as ONNX files? My use case is embedded pre-trained model inference. I don’t necessarily need the raw pytorch or tensorflow model. The default ONNX quantized export would be enough for me. I can do that myself using a combination of the huggingface download utils and the ONNX conversion script. Ideally, I would get the ONNX file directly from 🤗 [ADDITION] It would probably not be as simple as “get the ONNX file” if we need a plug-and-play experience. Happy to brainstorm on the right format! Cheers, Alex Combessie ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 14, 2020, 8:29am UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/6 "2020-10-14T08:29:35Z")
Hi @dataiku, give this a try [https://github.com/patil-suraj/onnx\_transformers](https://github.com/patil-suraj/onnx_transformers) Note: this is a utility project to allow easy inference, which creates the grapth on the fly. Pinging @julien-c to see if it’s possible to host onnx files on hub. ---
**Author:** ![dataiku](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dataiku/32/2634_2.png) [@dataiku](https://discuss.huggingface.co/u/dataiku)\ **Post date:** [October 14, 2020, 9:11am UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/7 "2020-10-14T09:11:01Z")
Hi @valhalla, Just had a look. I am exactly looking for that kind of simplicity: `nlp = pipeline("sentiment-analysis", onnx=True)` 🙂 Good prototype. The main caveats I can think of are: (i) ONNX conversion is done on-device - I’d rather pull pre-computed ONNX files from the model hub (ii) some level duplication of huggingface source code (iii) possible improvements in the tokenizer choices to use fast versions when available I would really like to have this option native in huggingface so I can use it in production applications when inference speed matters a lot. Cheers, Alex ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 14, 2020, 9:15am UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/8 "2020-10-14T09:15:46Z")
i) we could pull if they are available ii) Duplicated to provide the same API iii) Easy to add. Will add it. HF is considering this in the pipeline V2 see [[RFC] Transformers Pipeline v2](https://discuss.huggingface.co/t/rfc-transformers-pipeline-v2/1209) ---
**Author:** ![dataiku](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dataiku/32/2634_2.png) [@dataiku](https://discuss.huggingface.co/u/dataiku)\ **Post date:** [October 14, 2020, 9:46am UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/9 "2020-10-14T09:46:40Z")
Thanks for pointing it out to [[RFC] Transformers Pipeline v2](https://discuss.huggingface.co/t/rfc-transformers-pipeline-v2/1209). That’s good to know. For (ii) is it possible to leverage Python class inheritance features to only add/override the methods which are necessary for ONNX? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 14, 2020, 9:48am UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/10 "2020-10-14T09:48:24Z")
definitely possible! Copied everything so that I can have full control ---
**Author:** ![julien-c](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/julien-c/32/41937_2.png) [@julien-c](https://discuss.huggingface.co/u/julien-c)\ **Post date:** [October 15, 2020, 5:23pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/11 "2020-10-15T17:23:58Z")
We do have a few model in onnx on the hub, and we support it as a searchable filetype: [https://huggingface.co/models?filter=onnx](https://huggingface.co/models?filter=onnx) (cc @mfuntowicz) We’ll look into auto-converting models to onnx in the future. For now, feel free to ask model authors (or us) to upload converted files manually ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 16, 2020, 5:17pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/12 "2020-10-16T17:17:18Z")
If we upload onnx models, will they appear in `onnx` filter ? ---
**Author:** ![ierezell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ierezell/32/2517_2.png) [@ierezell](https://discuss.huggingface.co/u/ierezell)\ **Post date:** [February 12, 2021, 8:19pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/13 "2021-02-12T20:19:41Z")
For now there is only two models at [https://huggingface.co/models?filter=onnx](https://huggingface.co/models?filter=onnx). And I agree with @bdalal, it’s “quite easy” to convert a model to an onnx checkpoint and use it in let’s say NodeJs which don’t have a mature ecosystem for Deeplearning. I would convert them it if I need to fintune a model but for hub models it would be more convenient to download them directly (if converted automatically). I almost envisaged to convert the models I use and host them for my team to use… Also a `model.save(args, onnx=True)` to save a onnx file locally (by calling `convert_graph_to_onnx`) as well would be really useful ! So here is the 2cents of a really happy user 🙂 Thanks for this wonderfull tool ! Have a great day ---
**Author:** ![sam-writer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sam-writer/32/2607_2.png) [@sam-writer](https://discuss.huggingface.co/u/sam-writer)\ **Post date:** [August 30, 2021, 6:56pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/14 "2021-08-30T18:56:59Z")
Is it possible to use ONNX models directly from the Hub, either by directly referencing the file, or via the `from_pretrained()` method? That is, before converting to ONNX, I instantiate a (private) model via `model = AutoModelForSeq2SeqLM.from_pretrained('org/model_name', use_auth_token=True)`. If I push a `.onnx` file up to `'org/model_name'`, how can I load it? ---
**Author:** ![julien-c](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/julien-c/32/41937_2.png) [@julien-c](https://discuss.huggingface.co/u/julien-c)\ **Post date:** [September 1, 2021, 9:40am UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/15 "2021-09-01T09:40:54Z")
@mfuntowicz @lysandre or @Narsil might know! ---
**Author:** ![Narsil](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/narsil/32/40029_2.png) [@Narsil](https://discuss.huggingface.co/u/Narsil)\ **Post date:** [September 1, 2021, 1:41pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/16 "2021-09-01T13:41:04Z")
Hi @sam-writer, there’s currently no way to do that with `from_pretrained`, and I don’t think it will be in the foreseeable future (first of all we would have to choose a runtime for that onnx file, and using onnxruntime forces a lot onto the library, for which transformers is probably not the best place to do). That being said, if you have a `.onnx` file on your repo, the best way to use it is with you preferred runtime (onnxruntime most likely). ```python from huggingface_hub import cached_download file = cached_download("https://huggingface.co/{USERNAME}/{MODEL_ID}/raw/main/{FILE}.onnx", use_auth_token=os.getenv("API_TOKEN")) sess = InferenceSession(file) # go on with inferencing with onnx ``` Does that help ? ---
**Author:** ![sam-writer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sam-writer/32/2607_2.png) [@sam-writer](https://discuss.huggingface.co/u/sam-writer)\ **Post date:** [September 1, 2021, 4:48pm UTC](https://discuss.huggingface.co/t/supporting-onnx-optimized-models/891/17 "2021-09-01T16:48:03Z")
YES! Thank you Narsil, I didn’t know about `cached_download`, nor how the file structure worked with `/raw/main`. This is great, thanks so much.