# Support for exporting generate function to ONNX? **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 10, 2022, 8:52pm UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501 "2022-08-10T20:52:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![nifarn](https://avatars.discourse-cdn.com/v4/letter/n/82dd89/32.png) [@nifarn](https://discuss.huggingface.co/u/nifarn)\ **Post date:** [August 10, 2022, 8:52pm UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501/1 "2022-08-10T20:52:54Z")
I was wondering if huggingface provided any support for exporting the generate function from the `transformers` library to ONNX? Mainly, I was trying to create an ONNX model using a GPT2 style transformer in order to speed up inference when generating replies to a conversation. I see there’s some support for exporting a single call to GPT2, but not the entire for loop used in greedy decoding/beam search/nucleus sampling etc. ---
**Author:** ![guillermogabrielli](https://avatars.discourse-cdn.com/v4/letter/g/b19c9b/32.png) [@guillermogabrielli](https://discuss.huggingface.co/u/guillermogabrielli)\ **Post date:** [August 11, 2022, 12:14am UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501/2 "2022-08-11T00:14:20Z")
Take a look at huggingface [optimum](https://github.com/huggingface/optimum), but not all models are supported. [Documentation.](https://huggingface.co/docs/optimum/onnxruntime/modeling_ort#optimum.onnxruntime.ORTModelForCausalLM) ---
**Author:** ![nifarn](https://avatars.discourse-cdn.com/v4/letter/n/82dd89/32.png) [@nifarn](https://discuss.huggingface.co/u/nifarn)\ **Post date:** [August 11, 2022, 6:43pm UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501/3 "2022-08-11T18:43:40Z")
Thanks for the response @guillermogabrielli. I’ve already taken a cursory look at optimum, however all the export scenarios seem pretty targeted. I could not find a way to export an LM + generalized decoder. ---
**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\ **Post date:** [December 7, 2022, 6:11pm UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501/4 "2022-12-07T18:11:33Z")
Hi @nifarn , could you elaborate on what you would like to see that is not available in Optimum for encoder-decoder models? ---
**Author:** ![in-certo](https://avatars.discourse-cdn.com/v4/letter/i/c6cbf5/32.png) [@in-certo](https://discuss.huggingface.co/u/in-certo)\ **Post date:** [February 2, 2023, 11:00am UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501/5 "2023-02-02T11:00:59Z")
Hi @fxmarty I have the same demand. I want to export both the model and the generation strategy. The model architecture is similar to GPT with slight modifications. And the generation strategy, including greedy and beam search, requires pytorch to trace `if` and `for`. I found [a demo to export BART along with BeamSearch into ONNX](https://github.com/huggingface/transformers/tree/main/examples/research_projects/onnx/summarization). But it’s outdated. ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/00fda5f95e0d3efd74aa16998f637293d2d8b0e1.png) Any suggestion on how to achieve this in Optimum? @nifarn Could you share you final solution? Any help is appreciated! ---
**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\ **Post date:** [February 6, 2023, 12:10pm UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501/6 "2023-02-06T12:10:32Z")
Hi @in-certo , this is a very reasonable feature request indeed. The issue is tracked at [Make ORTModel PyTorch free · Issue #526 · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/issues/526) I very much doubt that transformers `generate()` will ever be jit scriptable. Therefore, the solution would probably be to rewrite `generate()` and greedy and beam search for them to be jit.scriptable. The other option is to use ONNX Runtime `GreedySearch` and `BeamSearch` ops, but those are usable only with CPUExecutionProvider and CUDAExecutionProvider (useless if you want to use TensorRT for example). You could torch.jit.trace the model itself, and then torch.jit.script the generate method. ---
**Author:** ![in-certo](https://avatars.discourse-cdn.com/v4/letter/i/c6cbf5/32.png) [@in-certo](https://discuss.huggingface.co/u/in-certo)\ **Post date:** [February 8, 2023, 7:14am UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501/7 "2023-02-08T07:14:36Z")
Hi @fxmarty Thanks for your suggestion. I have some questions on how to convert a generation model to TorchScript. Take GLM as an example. The signature of its forward function contains an Optional[Tensor] variable `mem`. The code can be found here: [modeling\_glm.py · BAAI/glm-10b at main (huggingface.co)](https://huggingface.co/BAAI/glm-10b/blob/main/modeling_glm.py#L890). If we use `torch.jit.trace` function to convert it to TorchScript. How can we trace two different forward passes? (One takes a real `mem` variable which is a List[Tensor], and the other takes a `None`) ---
**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\ **Post date:** [February 8, 2023, 12:56pm UTC](https://discuss.huggingface.co/t/support-for-exporting-generate-function-to-onnx/21501/8 "2023-02-08T12:56:24Z")
For generation, I _think_ you will not be able to obtain a meaningful ScriptModule using `torch.jit.trace`. This is because the number of generation steps and early stopping depend on which tokens are generated, the number of loops is dynamic. `torch.jit.trace` would just hardcode the number of loops from the sample example provided during tracing, but this is not useful for other samples. > If we use `torch.jit.trace` function to convert it to TorchScript. How can we trace two different forward passes? Normally you should just provide the model with a different set of inputs that trigger the controlflows you would like to use.