# Pegasus Inference for production usecase **URL:** \ **Category:** Beginners\ **Created:** [December 3, 2020, 5:02pm UTC](https://discuss.huggingface.co/t/pegasus-inference-for-production-usecase/2486 "2020-12-03T17:02:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![karthikgali](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@karthikgali](https://discuss.huggingface.co/u/karthikgali)\ **Post date:** [December 3, 2020, 5:02pm UTC](https://discuss.huggingface.co/t/pegasus-inference-for-production-usecase/2486/1 "2020-12-03T17:02:28Z")
Hi, We have finetuned [distill-pegasus-cnn-16-4](https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4) summarization model on our own data and results look good. However, when we want to deploy it for a real-time production use case - it is taking huge time on ml.c5.xlarge CPU (around 13seconds per document in a sequence). We tried a [g4dn.xlarge](https://aws.amazon.com/ec2/instance-types/g4/) GPU for inference and it is taking around 1.7seconds for one document in a sequence. Inference on a GPU is a costly affair. Could anyone suggest me if anyone have any ideas to make faster with less cost. I have tried “onnx runtime”, [TorchScript](https://huggingface.co/transformers/serialization.html#saving-a-model) but both of them dont have support for pegasus model. Do we have any timelines for supporting pegasus. I tried num\_beams as well - above numbers are after using them. **Inference code:** import torch from transformers import PegasusForConditionalGeneration, PegasusTokenizer src\_text = [ “”" PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.""" ] model\_name = “sshleifer/distill-pegasus-cnn-16-4” torch\_device = ‘cuda’ if torch.cuda.is\_available() else ‘cpu’ tokenizer = PegasusTokenizer.from\_pretrained(model\_name) model = PegasusForConditionalGeneration.from\_pretrained(model\_name).to(torch\_device) inputs = tokenizer.batch\_encode\_plus(src\_text, truncation=True, padding=“longest”, return\_tensors=“pt”).to(torch\_device) translated = model.generate(inputs[“input\_ids”], num\_beams=2) output = [tokenizer.decode(g, skip\_special\_tokens=True, clean\_up\_tokenization\_spaces=False) for g in translated][0] print(output) Thanks in advance for your help. Apologies if I am missing something obvious. Regards, Karthik ---
**Author:** ![karthikgali](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@karthikgali](https://discuss.huggingface.co/u/karthikgali)\ **Post date:** [December 16, 2020, 12:09pm UTC](https://discuss.huggingface.co/t/pegasus-inference-for-production-usecase/2486/2 "2020-12-16T12:09:48Z")
Hi, Could someone help in this regard? Thanks in advance, Karthik ---
**Author:** ![FL33TW00D](https://avatars.discourse-cdn.com/v4/letter/f/e0b2c6/32.png) [@FL33TW00D](https://discuss.huggingface.co/u/FL33TW00D)\ **Post date:** [December 16, 2020, 8:06pm UTC](https://discuss.huggingface.co/t/pegasus-inference-for-production-usecase/2486/3 "2020-12-16T20:06:28Z")
HI Karthik, Forgive me if you have looked into this, but have you tried model quantization? > **[Quantization — PyTorch 1.7.0 documentation](https://pytorch.org/docs/stable/quantization.html)** If ONNX is not supported, then your only other bet is to decrease the size of the model or change some parameters like you’ve been trying. ---
**Author:** ![clem](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/clem/32/2838_2.png) [@clem](https://discuss.huggingface.co/u/clem)\ **Post date:** [December 17, 2020, 3:28am UTC](https://discuss.huggingface.co/t/pegasus-inference-for-production-usecase/2486/4 "2020-12-17T03:28:28Z")
Hi Karthik, one option would be to serve it with our hosted inference API for example with our startup plan here: [https://huggingface.co/pricing](https://huggingface.co/pricing). We would take care of the optimizations for you. If you have any question, feel free to write an email to [api-enterprise@huggingface.co](mailto:api-enterprise@huggingface.co). ---
**Author:** ![karthikgali](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@karthikgali](https://discuss.huggingface.co/u/karthikgali)\ **Post date:** [December 20, 2020, 7:46am UTC](https://discuss.huggingface.co/t/pegasus-inference-for-production-usecase/2486/5 "2020-12-20T07:46:09Z")
Thanks for your inputs. I will check them. ---
**Author:** ![the-pale-king](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@the-pale-king](https://discuss.huggingface.co/u/the-pale-king)\ **Post date:** [December 20, 2020, 2:42pm UTC](https://discuss.huggingface.co/t/pegasus-inference-for-production-usecase/2486/6 "2020-12-20T14:42:16Z")
@karthikgali Something is confusing about this post for me. You wrote > We have finetuned [distill-pegasus-cnn-16-4](https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4) But I can see in your code that you are loading `model_name = “sshleifer/distill-pegasus-cnn-16-4”` If you fine-tuned it, wouldn’t you load your fine-tuned model, and not the model that has already been tuned by `sshleifer`? Put another way, see here: [Pegasus](https://huggingface.co/transformers/model_doc/pegasus.html#checkpoints) > All the [checkpoints](https://huggingface.co/models?search=pegasus) are fine-tuned for summarization, besides pegasus-large, whence the other checkpoints are fine-tuned My understanding from this is that, if we, as the newbie user, have some data we want to use with Pegasus, we should do this: 1. Start with pegasus-large: [google/pegasus-large · Hugging Face](https://huggingface.co/google/pegasus-large) 2. Fine tune it on our own data 3. Use the `pytorch_model.bin` output from this fine tuning process to run inference on our own data. Note – I am not suggesting you are doing anything wrong. I’m just working with pegasus also, and reading your message makes me think that I’m maybe getting something wrong. Also – I am finding, when fine-tuning Pegasus, using `pegasus-large`, that the RAM requirements for even just a `batch size` of `1` are so extreme, that a Nvidia card with `16GB` of memory is required… just to run the `batch size` of `1`! So at this point I am thinking that maybe my training will run better on the CPU, using a machine with a huge amount of ram… like `512GB` of ram… as this seems to allow a much bigger batch size, like up to `64` or `128`. ---
**Author:** ![the-pale-king](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@the-pale-king](https://discuss.huggingface.co/u/the-pale-king)\ **Post date:** [February 26, 2021, 1:28am UTC](https://discuss.huggingface.co/t/pegasus-inference-for-production-usecase/2486/7 "2021-02-26T01:28:13Z")
@karthikgali Did you end up with a decent solution for this issue? Did you try the solution suggested on StackOverflow below? Here are some leads: > > > I tried to convert Pegasus to the ONNX format using \[this \](https://colab.resear…ch.google.com/github/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb#scrollTo=foYlXrSksR\_v) guide but it failed. Can Pegasus be converted to the ONNX format or is that not possible yet? > > > > It seems like it's not available now, I got this error: > \`Error while converting… the model: Unrecognized configuration class \ for this kind of AutoModel: AutoModel. > Model type should be one of RetriBertConfig, T5Config, DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, BartConfig, LongformerConfig, RobertaConfig, LayoutLMConfig, SqueezeBertConfig, BertConfig, OpenAIGPTConfig, GPT2Config, MobileBertConfig, TransfoXLConfig, XLNetConfig, FlaubertConfig, FSMTConfig, XLMConfig, CTRLConfig, ElectraConfig, ReformerConfig, FunnelConfig, LxmertConfig, BertGenerationConfig, DebertaConfig, DPRConfig, XLMProphetNetConfig, ProphetNetConfig.\` > > Which is fair since pegasus is a new addition. Is it something the team plans to do soon? > > Or can someone point me some resources on if there are other ways to export a pre-trained model from huggingface? I'm pretty new to the machine learning thing :p > > Thanks all! thanks