# How to generate on multiple GPU's

**URL:** <https://discuss.huggingface.co/t/how-to-generate-on-multiple-gpus/21760>\
**Category:** Intermediate\
**Created:** [August 17, 2022, 8:40pm UTC](https://discuss.huggingface.co/t/how-to-generate-on-multiple-gpus/21760 "2022-08-17T20:40:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Samuel-Fipps](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Samuel-Fipps](https://discuss.huggingface.co/u/Samuel-Fipps)\
**Post date:** [August 17, 2022, 8:40pm UTC](https://discuss.huggingface.co/t/how-to-generate-on-multiple-gpus/21760/1 "2022-08-17T20:40:21Z")

</div>

The last time this question was asked it was in 2020, and I can’t seem to how out how to do this. Is it possible?

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [August 19, 2022, 2:44pm UTC](https://discuss.huggingface.co/t/how-to-generate-on-multiple-gpus/21760/2 "2022-08-19T14:44:29Z")

</div>

The bottleneck of generation is the model forward pass, so being able to run the model forward pass in multiple GPUs should do it.

I’m not knowledgeable about multi-GPU inference, especially in PyTorch, maybe @sgugger knows how to do it

(the answer will also be useful to me 🙂 )

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [August 21, 2022, 9:29pm UTC](https://discuss.huggingface.co/t/how-to-generate-on-multiple-gpus/21760/3 "2022-08-21T21:29:53Z")

</div>

Check out [FastTransformer](https://developer.nvidia.com/blog/accelerated-inference-for-large-transformer-models-using-nvidia-fastertransformer-and-nvidia-triton-inference-server/), a library focusing on large models, spanning many GPUs and nodes in a distributed manner.

---

<div class="post-metadata">

**Author:** ![MikeyBeez](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mikeybeez/32/21067_2.png) [@MikeyBeez](https://discuss.huggingface.co/u/MikeyBeez)\
**Post date:** [August 30, 2022, 4:48am UTC](https://discuss.huggingface.co/t/how-to-generate-on-multiple-gpus/21760/4 "2022-08-30T04:48:24Z")

</div>

I think you can use a pip module called accelerate. I believe this module can also train on multiple machines. I don’t know if they need to be heterogeneous. I use it to switch from CPU to MPS on my Mac. It has options for multi-gpu.