# Generate() method for models converted to torchscript

**URL:** <https://discuss.huggingface.co/t/generate-method-for-models-converted-to-torchscript/47728>\
**Category:** Models\
**Created:** [July 22, 2023, 10:14am UTC](https://discuss.huggingface.co/t/generate-method-for-models-converted-to-torchscript/47728 "2023-07-22T10:14:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![botteaap](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/botteaap/32/17832_2.png) [@botteaap](https://discuss.huggingface.co/u/botteaap)\
**Post date:** [July 22, 2023, 10:14am UTC](https://discuss.huggingface.co/t/generate-method-for-models-converted-to-torchscript/47728/1 "2023-07-22T10:14:43Z")

</div>

I’ve been experimenting exporting git-large-coco to torchscript and with a minor adjustment to the transformers library this seems to work. However `GitForCausalLM` uses the `generate()` function, not just just a plain model() invocation, so I’m stuck on how to use the torchscript version of the model.

Am I correct that I would need to reimplement whatever generate() is doing if I want to use the torchscript model? I’d eventually like to run it in a non-python environment, but for now I’m only experimenting.

(edit: I did find [Generate without using the generate method](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379) which give me some clues, but I’m not getting the correct output yet)

Can I also load the torchscript model back into transformers just to see if the model behaves the same?

---

<div class="post-metadata">

**Author:** ![Ujan](https://avatars.discourse-cdn.com/v4/letter/u/6de8d8/32.png) [@Ujan](https://discuss.huggingface.co/u/Ujan)\
**Post date:** [August 1, 2023, 6:58am UTC](https://discuss.huggingface.co/t/generate-method-for-models-converted-to-torchscript/47728/2 "2023-08-01T06:58:12Z")

</div>

Were you able to make any progress on this? I tried implementing the code from the post you have linked. But it is much slower than the generate( ) method.

---

<div class="post-metadata">

**Author:** ![botteaap](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/botteaap/32/17832_2.png) [@botteaap](https://discuss.huggingface.co/u/botteaap)\
**Post date:** [August 1, 2023, 8:04am UTC](https://discuss.huggingface.co/t/generate-method-for-models-converted-to-torchscript/47728/3 "2023-08-01T08:04:17Z")

</div>

No not really. I did try to implement it but I think the model I’m using is not using greedy matching or something and the results weren’t similar. Still not sure why though.

Then I found out I messed up the torchscript conversion and that it’s probably not going to work for what I’m trying to do, so I didn’t pursue it further.