# Multi-gpu inference

**URL:** <https://discuss.huggingface.co/t/multi-gpu-inference/86567>\
**Category:** Beginners\
**Created:** [May 13, 2024, 11:59pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference/86567 "2024-05-13T23:59:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![somesaba](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@somesaba](https://discuss.huggingface.co/u/somesaba)\
**Post date:** [May 13, 2024, 11:59pm UTC](https://discuss.huggingface.co/t/multi-gpu-inference/86567/1 "2024-05-13T23:59:53Z")

</div>

Hello, I am trying to maximize inference speed of a single prompt on a small (7B) model. I have a server with 4 GPUs. It seems possible to use accelerate to speed up inference. Does anyone have example code? I only see examples of splitting multiple prompts across GPUs but I only have 1 prompt at a time.

Thank you.

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [May 14, 2024, 12:26am UTC](https://discuss.huggingface.co/t/multi-gpu-inference/86567/2 "2024-05-14T00:26:52Z")

</div>

If your model fits on a single GPU, that will always be faster than multiple GPUs as doing so would involve communication bandwidth and slow things down some

---

<div class="post-metadata">

**Author:** ![somesaba](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@somesaba](https://discuss.huggingface.co/u/somesaba)\
**Post date:** [May 14, 2024, 1:01am UTC](https://discuss.huggingface.co/t/multi-gpu-inference/86567/3 "2024-05-14T01:01:42Z")

</div>

Thank you. Is there any other way to improve inference speed? I’m already using an H100 with 4-bit quantization and flash attention 2.