# Multi-GPU is slower than single GPU when running examples

**URL:** <https://discuss.huggingface.co/t/multi-gpu-is-slower-than-single-gpu-when-running-examples/96885>\
**Category:** 🤗Accelerate\
**Created:** [July 11, 2024, 8:17pm UTC](https://discuss.huggingface.co/t/multi-gpu-is-slower-than-single-gpu-when-running-examples/96885 "2024-07-11T20:17:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![duncanswilson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/duncanswilson/32/29231_2.png) [@duncanswilson](https://discuss.huggingface.co/u/duncanswilson)\
**Post date:** [July 11, 2024, 8:17pm UTC](https://discuss.huggingface.co/t/multi-gpu-is-slower-than-single-gpu-when-running-examples/96885/1 "2024-07-11T20:17:43Z")

</div>

I have a machine with 3 3090s and have been using accelerate with [lm\_eval](https://github.com/EleutherAI/lm-evaluation-harness/) to speed up inference and seeing sensible results.

I wrote some custom training scripts using accelerate but noticed about a 3x slowdown vs the single GPU case. While debugging, I decided to try the `nlp_example.py` and I’m getting significant slowdowns between the single GPU and multi-GPU case there too.

I’ve found this [topic](https://discuss.huggingface.co/t/single-gpu-is-faster-than-multiple-gpus/71383) on a similar slowdown, but I am able to see performance gains with accelerate and lm\_eval, so I doubt that it is a CUDA/pytorch version incompatibility…

For reference, when running `nlp_example.py` on one GPU I am getting 44 secs total for three epochs and for the multiGPU case I get 4 mins.

I’m happy to provide any other information (package versions and CUDA versions) if that is needed.

---

<div class="post-metadata">

**Author:** ![duncanswilson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/duncanswilson/32/29231_2.png) [@duncanswilson](https://discuss.huggingface.co/u/duncanswilson)\
**Post date:** [July 24, 2024, 7:33pm UTC](https://discuss.huggingface.co/t/multi-gpu-is-slower-than-single-gpu-when-running-examples/96885/2 "2024-07-24T19:33:21Z")

</div>

If anyone else runs into this issue: it turned out to be a BIOS level change that was needed in order to fix the communication overhead. Specifically changing the link speed on the PCI ports from Gen 1 to Gen 4 resulted in seeing speedups using multiple GPUs for fine-tuning!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [July 25, 2024, 7:33am UTC](https://discuss.huggingface.co/t/multi-gpu-is-slower-than-single-gpu-when-running-examples/96885/3 "2024-07-25T07:33:26Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.