# How to parallelize model.generate?

**URL:** <https://discuss.huggingface.co/t/how-to-parallelize-model-generate/3365>\
**Category:** 🤗Transformers\
**Created:** [January 24, 2021, 11:15pm UTC](https://discuss.huggingface.co/t/how-to-parallelize-model-generate/3365 "2021-01-24T23:15:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![berkayberabi](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@berkayberabi](https://discuss.huggingface.co/u/berkayberabi)\
**Post date:** [January 24, 2021, 11:15pm UTC](https://discuss.huggingface.co/t/how-to-parallelize-model-generate/3365/1 "2021-01-24T23:15:24Z")

</div>

Hi,

I am using model.generate() on the T5 model. I have a for loop where I iterate over the samples one by one. As expected, it is too slow. I know that trainer class has trainer.predict() method but then I can not apply beam decoding right?

I also know that I can pass batches to tokenizer and model.generate() and speed up the training but then still I will have to have a for loop where I iterate over the batches, right?

Is there a way to parallize the generation while using beam strategy and other stuff that model.generate provides?

---

<div class="post-metadata">

**Author:** ![sgonzalezsilot](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgonzalezsilot/32/22042_2.png) [@sgonzalezsilot](https://discuss.huggingface.co/u/sgonzalezsilot)\
**Post date:** [September 7, 2022, 7:36am UTC](https://discuss.huggingface.co/t/how-to-parallelize-model-generate/3365/2 "2022-09-07T07:36:58Z")

</div>

Hi,

Did you find any good solution?

Thanks.