# Language generation with torchscript model? **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 11, 2020, 2:32am UTC](https://discuss.huggingface.co/t/language-generation-with-torchscript-model/669 "2020-08-11T02:32:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![laphangho](https://avatars.discourse-cdn.com/v4/letter/l/d07c76/32.png) [@laphangho](https://discuss.huggingface.co/u/laphangho)\ **Post date:** [August 11, 2020, 2:32am UTC](https://discuss.huggingface.co/t/language-generation-with-torchscript-model/669/1 "2020-08-11T02:32:44Z")
I have fine-tuned a summarization model following the Hugging Face seq2seq guide (starting from sshleifer/distilbart-xsum-12-6). Our team is interested in using AWS elastic inference for deployment for cost reduction. (e.g. similar to this [https://aws.amazon.com/blogs/machine-learning/fine-tuning-a-pytorch-bert-model-and-deploying-it-with-amazon-elastic-inference-on-amazon-sagemaker/](https://aws.amazon.com/blogs/machine-learning/fine-tuning-a-pytorch-bert-model-and-deploying-it-with-amazon-elastic-inference-on-amazon-sagemaker/)) I was wondering whether there’s any examples or any suggested way to use the beam searching logic in BartForConditionalGeneration with model inference from a torchscript model. Most of the examples for torchscript I’ve found are with classification tasks where this isn’t necessary. ---
**Author:** ![setu4993](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/setu4993/32/26_2.png) [@setu4993](https://discuss.huggingface.co/u/setu4993)\ **Post date:** [September 15, 2020, 1:14am UTC](https://discuss.huggingface.co/t/language-generation-with-torchscript-model/669/2 "2020-09-15T01:14:36Z")
I’ve had success with deploying a BartForConditionalGeneration model using SageMaker with EI. Try: ```python model = BartForConditionalGeneration.from_pretrained(model_dir, torchscript=True) ``` ---
**Author:** ![laphangho](https://avatars.discourse-cdn.com/v4/letter/l/d07c76/32.png) [@laphangho](https://discuss.huggingface.co/u/laphangho)\ **Post date:** [September 15, 2020, 1:52am UTC](https://discuss.huggingface.co/t/language-generation-with-torchscript-model/669/3 "2020-09-15T01:52:40Z")
Thanks a lot for that reply @setu4993! It looks really promising, we’ll give it a try ---
**Author:** ![setu4993](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/setu4993/32/26_2.png) [@setu4993](https://discuss.huggingface.co/u/setu4993)\ **Post date:** [September 15, 2020, 1:56am UTC](https://discuss.huggingface.co/t/language-generation-with-torchscript-model/669/4 "2020-09-15T01:56:52Z")
@[laphangho](https://discuss.huggingface.co/u/laphangho) Good luck! To add a little more context: SageMaker wants a ScriptModule, not trace. Trace is not possible with `.generate()`, but script works fine. And to use script mode, saving the model in a different way (than the default `.save_pretrained()` method is not required since `torchscript=True` can simply be provided as an additional argument when creating the model object. ---
**Author:** ![setu4993](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/setu4993/32/26_2.png) [@setu4993](https://discuss.huggingface.co/u/setu4993)\ **Post date:** [November 17, 2020, 6:54am UTC](https://discuss.huggingface.co/t/language-generation-with-torchscript-model/669/6 "2020-11-17T06:54:10Z")
Unfortunately, soon after I wrote that, I realized beam search (`.generate()`) was actually not working with EI (though `.forward()` does, but using that means losing out on beam search). I faced the same situation whereby the model inference was taking place on CPU only, no EI, so had to eventually switch up to a GPU instance. That was ~2 months ago now, so don’t know if something has changed since I haven’t looked back. A couple other things: 1. As of writing (and this has been the same for a few months now) EI on PyTorch is only [supported on PyTorch 1.3.1](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#elastic-inference-containers). So, if using any other version of PyTorch, it will fallback to CPU. 2. Since PyTorch 1.6 the way in which the models are [saved has changed](https://pytorch.org/docs/stable/generated/torch.save.html). To make a model backward compatible, it might have to be saved with `_use_new_zipfile_serialization=False`. Sorry I couldn’t help more. Please do share if you find other workarounds here. Thanks! ---
**Author:** ![Ankita](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@Ankita](https://discuss.huggingface.co/u/Ankita)\ **Post date:** [April 25, 2021, 12:39pm UTC](https://discuss.huggingface.co/t/language-generation-with-torchscript-model/669/7 "2021-04-25T12:39:02Z")
> [@artmatsak](#): > > se Elastic Inference? Has anyone figured out a way to run inference (same as the .generate() method) for seq-to-seq models on Elastic Inference? I am trying to run inference for Seq-to-Seq models (like BART, Pegasus) on Elastic Inference with EC2. So far, I have been able to use the TorchScript example (1) and store the model, but unable to figure out how to run the inference on it. (1) [Export to ONNX](https://huggingface.co/transformers/serialization.html#using-a-traced-model-for-inference) ---
**Author:** ![derekzen](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@derekzen](https://discuss.huggingface.co/u/derekzen)\ **Post date:** [November 20, 2021, 1:08pm UTC](https://discuss.huggingface.co/t/language-generation-with-torchscript-model/669/8 "2021-11-20T13:08:17Z")
How did you script the `generate` method? I use `torch.jit.script(model.generate)` and got and error: `NotSupportedError: Compiled functions can't take variable number of arguments or use keyword-only arguments with defaults: ...`